Oct 30 02:41:18.476474 ip-10-0-128-241 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:41:18.911215 ip-10-0-128-241 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:41:18.911215 ip-10-0-128-241 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:41:18.911215 ip-10-0-128-241 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:41:18.911215 ip-10-0-128-241 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:41:18.915506 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915438    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:18.915506 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915506    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:18.915575 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915510    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:18.915575 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915514    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:18.915575 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915517    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:18.915575 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915521    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:18.915575 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915526    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:18.915575 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915529    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:18.915575 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915532    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:18.915575 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915535    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:18.915575 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915538    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:18.915575 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915540    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:18.915575 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915543    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:18.915575 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915546    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:18.915575 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915549    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:18.915575 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915551    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:18.915575 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915554    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:18.915575 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915556    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:18.915575 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915561    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:18.915575 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915563    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:18.915575 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915566    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915569    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915572    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915574    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915578    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915581    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915585    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915588    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915598    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915601    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915604    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915606    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915609    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915611    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915614    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915617    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915619    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915622    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915624    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915627    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:18.916014 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915630    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915632    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915635    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915637    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915640    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915643    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915645    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915648    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915650    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915653    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915656    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915658    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915662    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915665    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915668    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915671    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915673    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915676    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915679    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915682    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:18.916504 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915684    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:18.916970 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915687    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:18.916970 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915689    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:18.916970 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915692    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:18.916970 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915694    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:18.916970 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915697    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:18.916970 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915700    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:18.916970 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915702    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:18.916970 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915705    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:18.916970 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915707    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:18.916970 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915710    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:18.916970 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915712    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:18.916970 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915715    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:18.916970 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915718    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:18.916970 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915720    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:18.916970 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915723    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:18.916970 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915726    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:18.916970 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915729    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:18.916970 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915733    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:18.916970 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915735    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:18.917431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915738    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:18.917431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915741    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:18.917431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915744    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:18.917431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915746    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:18.917431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915749    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:18.917431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915752    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:18.917431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915754    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:18.917431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915757    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:18.917431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915760    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:18.917431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915762    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:18.917431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915765    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:18.917431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915768    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:18.917431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915771    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:18.917431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915773    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:18.917431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915776    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:18.917431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915780    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:18.917431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915784    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:18.917431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915787    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:18.917431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915790    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:18.917892 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915793    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:18.917892 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915795    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:18.917892 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915799    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:18.917892 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915802    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:18.917892 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915804    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:18.917892 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915807    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:18.917892 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.915810    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:18.917892 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916592    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:18.917892 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916599    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:18.917892 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916602    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:18.917892 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916605    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:18.917892 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916608    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:18.917892 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916616    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:18.917892 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916619    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:18.917892 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916623    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:18.917892 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916626    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:18.917892 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916629    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:18.917892 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916631    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:18.917892 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916635    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916638    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916641    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916643    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916646    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916648    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916651    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916654    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916657    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916659    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916662    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916665    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916667    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916670    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916673    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916675    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916678    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916681    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916683    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916686    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:18.918366 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916689    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:18.918895 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916692    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:18.918895 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916695    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:18.918895 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916697    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:18.918895 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916700    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:18.918895 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916702    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:18.918895 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916705    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:18.918895 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916708    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:18.918895 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916711    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:18.918895 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916714    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:18.918895 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916717    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:18.918895 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916720    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:18.918895 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916723    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:18.918895 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916725    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:18.918895 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916728    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:18.918895 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916731    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:18.918895 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916736    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:18.918895 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916740    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:18.918895 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916742    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:18.918895 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916745    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916748    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916751    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916753    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916756    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916758    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916761    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916764    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916767    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916769    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916772    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916775    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916777    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916780    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916782    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916785    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916788    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916790    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916793    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916795    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:18.919358 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916798    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:18.919839 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916802    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:18.919839 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916804    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:18.919839 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916807    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:18.919839 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916810    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:18.919839 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916813    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:18.919839 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916816    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:18.919839 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916819    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:18.919839 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916821    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:18.919839 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916824    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:18.919839 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916826    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:18.919839 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916829    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:18.919839 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916831    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:18.919839 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916834    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:18.919839 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916836    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:18.919839 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916839    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:18.919839 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916841    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:18.919839 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916844    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:18.919839 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916847    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:18.919839 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916850    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:18.920344 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916852    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:18.920344 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916855    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:18.920344 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916860    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:18.920344 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916864    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:18.920344 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916867    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:18.920344 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916870    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:18.920344 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916872    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:18.920344 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916875    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:18.920344 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916878    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:18.920344 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916880    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:18.920344 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916883    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:18.920344 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916885    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:18.920344 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916888    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:18.920344 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916891    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:18.920344 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916893    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:18.920344 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.916896    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:18.920344 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917373    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:18.920344 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917379    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:18.920344 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917382    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917384    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917387    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917390    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917393    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917396    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917398    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917402    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917404    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917407    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917410    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917413    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917416    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917419    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917421    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917424    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917427    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917430    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917433    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917435    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:18.920812 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917438    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917441    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917444    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917447    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917449    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917452    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917454    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917457    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917460    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917463    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917466    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917469    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917472    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917474    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917477    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917479    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917482    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917485    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917487    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917490    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:18.921303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917493    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917495    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917498    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917501    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917503    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917506    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917509    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917511    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917514    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917516    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917519    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917522    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917525    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917527    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917530    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917533    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917536    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917539    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917541    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917544    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:18.921777 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917547    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:18.922256 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917550    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:18.922256 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917553    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:18.922256 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917555    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:18.922256 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917558    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:18.922256 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917561    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:18.922256 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917564    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:18.922256 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917566    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:18.922256 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917569    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:18.922256 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917572    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:18.922256 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917574    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:18.922256 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917579    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:18.922256 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917582    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:18.922256 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917585    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:18.922256 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917587    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:18.922256 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917590    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:18.922256 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917592    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:18.922256 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917595    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:18.922256 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917598    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:18.922256 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917601    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:18.922701 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917603    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:18.922701 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917606    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:18.922701 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917609    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:18.922701 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917611    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:18.922701 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917614    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:18.922701 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917617    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:18.922701 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917619    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:18.922701 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917622    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:18.922701 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917625    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:18.922701 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917629    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:18.922701 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917632    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:18.922701 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917634    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:18.922701 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917637    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:18.922701 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917639    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:18.922701 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917643    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:18.922701 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917646    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:18.922701 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917648    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:18.922701 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917651    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:18.922701 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917654    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917656    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917659    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917661    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917664    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.917670    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917767    2578 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917775    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917785    2578 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917792    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917799    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917803    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917807    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917811    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917814    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917818    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917821    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917824    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917828    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917831    2578 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917834    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917837    2578 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917840    2578 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:41:18.923150 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917843    2578 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917869    2578 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917872    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917876    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917879    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917883    2578 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917888    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917891    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917894    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917898    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917901    2578 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917904    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917907    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917910    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917915    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917919    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917922    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917925    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917928    2578 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917932    2578 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917935    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917942    2578 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917945    2578 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917948    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:41:18.923710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917951    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917954    2578 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917958    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917961    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917964    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917968    2578 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917971    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917974    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917977    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917980    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917983    2578 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917986    2578 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917989    2578 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917993    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917996    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.917999    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918002    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918006    2578 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918009    2578 flags.go:64] FLAG: --help="false"
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918011    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-128-241.ec2.internal"
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918015    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918018    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918022    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918026    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:41:18.924357 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918029    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918032    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918035    2578 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918038    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918040    2578 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918043    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918046    2578 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918050    2578 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918053    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918056    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918058    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918061    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918064    2578 flags.go:64] FLAG: --lock-file=""
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918067    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918070    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918074    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918079    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918082    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918085    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918088    2578 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918091    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918094    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918097    2578 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918100    2578 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918105    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:41:18.924956 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918108    2578 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918111    2578 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918114    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918118    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918120    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918124    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918127    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918130    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918133    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918140    2578 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918143    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918146    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918149    2578 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918152    2578 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918155    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918158    2578 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918161    2578 flags.go:64] FLAG: --port="10250"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918164    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918167    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ecaa8c911d2983f9"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918170    2578 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918189    2578 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918192    2578 flags.go:64] FLAG: --register-node="true"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918195    2578 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918200    2578 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918203    2578 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918206    2578 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:41:18.925543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918208    2578 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918212    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918215    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918218    2578 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918221    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918224    2578 flags.go:64] FLAG: --runonce="false"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918228    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918231    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918234    2578 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918237    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918240    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918244    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918247    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918250    2578 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918253    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918256    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918259    2578 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918262    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918265    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918268    2578 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918271    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918276    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918279    2578 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918281    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918285    2578 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:41:18.926153 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918288    2578 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918292    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918295    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918298    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918301    2578 flags.go:64] FLAG: --v="2"
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918305    2578 flags.go:64] FLAG: --version="false"
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918309    2578 flags.go:64] FLAG: --vmodule=""
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918313    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918316    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918463    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918467    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918470    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918473    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918476    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918479    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918482    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918485    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918487    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918492    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918494    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918497    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:18.926748 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918500    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918502    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918505    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918508    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918511    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918513    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918516    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918519    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918521    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918524    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918527    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918529    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918533    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918536    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918539    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918541    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918544    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918546    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918549    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918551    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:18.927266 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918554    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918557    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918559    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918562    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918564    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918567    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918570    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918572    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918575    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918579    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918581    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918584    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918588    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918591    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918594    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918596    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918599    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918601    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918604    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918606    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:18.927735 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918609    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:18.928227 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918612    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:18.928227 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918614    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:18.928227 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918617    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:18.928227 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918621    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:18.928227 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918626    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:18.928227 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918630    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:18.928227 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918634    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:18.928227 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918637    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:18.928227 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918640    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:18.928227 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918643    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:18.928227 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918645    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:18.928227 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918648    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:18.928227 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918650    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:18.928227 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918653    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:18.928227 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918655    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:18.928227 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918658    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:18.928227 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918661    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:18.928227 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918663    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:18.928227 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918666    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:18.928703 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918669    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:18.928703 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918673    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:18.928703 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918676    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:18.928703 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918678    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:18.928703 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918681    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:18.928703 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918684    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:18.928703 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918686    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:18.928703 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918690    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:18.928703 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918695    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:18.928703 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918698    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:18.928703 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918701    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:18.928703 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918704    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:18.928703 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918707    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:18.928703 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918710    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:18.928703 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918713    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:18.928703 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918715    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:18.928703 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918719    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:18.928703 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918722    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:18.928703 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918724    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:18.929146 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918727    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:18.929146 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918730    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:18.929146 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918732    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:18.929146 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918735    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:18.929146 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918738    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:18.929146 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918740    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:18.929146 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918743    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:18.929146 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918745    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:18.929146 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918748    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:18.929146 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918751    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:18.929146 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918753    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:18.929146 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918756    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:18.929146 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918759    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:18.929146 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918761    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:18.929146 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918765    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:18.929146 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.918790    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:41:18.929542 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918827    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:18.929542 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918831    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:18.929542 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918834    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:18.929542 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918838    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:18.929542 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918840    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:18.929542 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918843    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:18.929542 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918846    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:18.929542 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918849    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:18.929542 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918851    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:18.929542 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918854    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:18.929542 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918857    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:18.929542 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918859    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:18.929542 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918862    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:18.929542 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918866    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:18.929542 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918869    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:18.929542 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918871    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:18.929542 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918874    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:18.929542 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918877    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:18.929542 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918880    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918883    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918885    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918888    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918890    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918893    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918896    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918899    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918901    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918904    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918907    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918910    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918913    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918917    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918919    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918922    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918925    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918927    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918930    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918932    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:18.930114 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918935    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918937    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918940    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918942    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918945    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918948    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918950    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918954    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918957    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918960    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918962    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918965    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918968    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918970    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918973    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918976    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918978    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918981    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918983    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918986    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:18.930882 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918988    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918991    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918994    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918996    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.918999    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919004    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919006    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919009    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919012    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919014    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919017    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919019    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919024    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919027    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919029    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919032    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919034    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919037    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919040    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919044    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:18.931381 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919047    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:18.931841 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919050    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:18.931841 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919053    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:18.931841 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919055    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:18.931841 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919058    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:18.931841 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919062    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:18.931841 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919065    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:18.931841 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919069    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:18.931841 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919072    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:18.931841 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919075    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:18.931841 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919078    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:18.931841 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919080    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:18.931841 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919083    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:18.931841 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919086    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:18.931841 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919088    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:18.931841 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919091    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:18.931841 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919093    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:18.931841 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919097    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:18.931841 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919100    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:18.931841 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919103    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:18.932303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919106    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:18.932303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919108    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:18.932303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919111    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:18.932303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919114    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:18.932303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919116    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:18.932303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919119    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:18.932303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919122    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:18.932303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.919126    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:18.932303 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.926138    2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:41:18.932303 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.926156    2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:41:18.932303 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.927721    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:41:18.932303 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.927728    2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:41:18.932303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927783    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:18.932303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927789    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:18.932303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927792    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:18.932303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927795    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:18.932303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927798    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:18.932303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927801    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:18.932303 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927804    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:18.932745 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927807    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:18.932745 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927810    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:18.932745 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927813    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:18.932745 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927816    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:18.932745 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927818    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:18.932745 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927821    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:18.932745 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927825    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:18.932745 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927828    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:18.932745 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927830    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:18.932745 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927833    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:18.932745 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927836    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:18.932745 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927838    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:18.932745 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927841    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:18.932745 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927844    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:18.932745 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927846    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:18.932745 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927849    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:18.932745 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927851    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:18.932745 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927854    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:18.932745 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927857    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:18.933288 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927860    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:18.933288 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927862    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:18.933288 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927865    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:18.933288 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927869    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:18.933288 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927872    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:18.933288 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927875    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:18.933288 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927878    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:18.933288 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927880    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:18.933288 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927883    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:18.933288 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927886    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:18.933288 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927888    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:18.933288 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927891    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:18.933288 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927893    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:18.933288 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927898    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:18.933288 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927902    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:18.933288 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927906    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:18.933288 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927909    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:18.933288 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927911    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:18.933288 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927914    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:18.933739 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927917    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:18.933739 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927920    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:18.933739 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927922    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:18.933739 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927925    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:18.933739 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927928    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:18.933739 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927931    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:18.933739 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927933    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:18.933739 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927936    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:18.933739 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927939    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:18.933739 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927941    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:18.933739 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927944    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:18.933739 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927947    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:18.933739 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927949    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:18.933739 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927952    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:18.933739 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927954    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:18.933739 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927957    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:18.933739 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927960    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:18.933739 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927968    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:18.933739 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927971    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:18.934173 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927974    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:18.934173 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927976    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:18.934173 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927979    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:18.934173 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927981    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:18.934173 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927985    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:18.934173 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927990    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:18.934173 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927993    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:18.934173 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927996    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:18.934173 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.927998    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:18.934173 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928001    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:18.934173 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928004    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:18.934173 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928006    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:18.934173 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928009    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:18.934173 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928012    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:18.934173 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928014    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:18.934173 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928017    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:18.934173 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928020    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:18.934173 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928023    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:18.934173 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928025    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928028    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928030    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928038    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928043    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928046    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928048    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928051    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928053    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928056    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928059    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928062    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928065    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928068    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928071    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928074    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928076    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928079    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928082    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928084    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:18.934656 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928087    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:18.935125 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928090    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:18.935125 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928092    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:18.935125 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928095    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:18.935125 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.928123    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:41:18.935125 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928289    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:18.935125 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928295    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:18.935125 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928298    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:18.935125 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928301    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:18.935125 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928304    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:18.935125 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928307    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:18.935125 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928310    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:18.935125 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928312    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:18.935125 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928315    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:18.935125 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928317    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:18.935125 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928320    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:18.935125 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928323    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928325    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928328    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928330    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928333    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928336    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928338    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928341    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928344    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928347    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928350    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928353    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928355    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928358    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928360    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928363    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928365    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928368    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928371    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928373    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:18.935510 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928376    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928379    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928382    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928384    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928387    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928389    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928392    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928394    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928397    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928400    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928402    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928405    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928408    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928410    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928413    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928416    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928418    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928421    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928424    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928426    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:18.935981 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928429    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:18.936471 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928432    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:18.936471 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928435    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:18.936471 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928439    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:18.936471 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928443    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:18.936471 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928446    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:18.936471 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928449    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:18.936471 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928452    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:18.936471 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928455    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:18.936471 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928457    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:18.936471 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928460    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:18.936471 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928463    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:18.936471 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928466    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:18.936471 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928468    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:18.936471 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928471    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:18.936471 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928474    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:18.936471 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928476    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:18.936471 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928479    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:18.936471 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928483    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:18.936880 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928486    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:18.936880 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928489    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:18.936880 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928492    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:18.936880 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928495    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:18.936880 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928497    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:18.936880 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928500    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:18.936880 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928503    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:18.936880 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928506    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:18.936880 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928508    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:18.936880 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928511    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:18.936880 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928513    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:18.936880 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928516    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:18.936880 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928519    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:18.936880 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928521    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:18.936880 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928524    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:18.936880 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928527    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:18.936880 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928529    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:18.936880 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928532    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:18.936880 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928535    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:18.937397 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928537    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:18.937397 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928540    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:18.937397 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928542    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:18.937397 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928545    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:18.937397 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928548    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:18.937397 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928550    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:18.937397 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928553    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:18.937397 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928555    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:18.937397 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928558    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:18.937397 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928561    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:18.937397 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928563    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:18.937397 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928566    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:18.937397 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928569    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:18.937397 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928571    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:18.937397 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928574    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:18.937397 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928577    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:18.937397 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928579    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:18.937787 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.928602    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:41:18.937787 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928632    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:18.937787 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928636    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:18.937787 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928640    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:18.937787 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928642    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:18.937787 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928645    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:18.937787 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928647    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:18.937787 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928650    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:18.937787 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928653    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:18.937787 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928656    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:18.937787 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928659    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:18.937787 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928661    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:18.937787 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928664    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:18.937787 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928666    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:18.937787 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928670    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:18.938138 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928674    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:18.938138 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928681    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:18.938138 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928685    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:18.938138 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928688    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:18.938138 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928690    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:18.938138 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928693    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:18.938138 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928696    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:18.938138 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928699    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:18.938138 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928701    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:18.938138 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928704    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:18.938138 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928707    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:18.938138 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928710    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:18.938138 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928712    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:18.938138 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928715    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:18.938138 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928718    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:18.938138 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928722    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:18.938138 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928725    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:18.938138 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928728    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:18.938566 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928731    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:18.938566 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928734    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:18.938566 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928737    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:18.938566 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928739    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:18.938566 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928742    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:18.938566 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928745    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:18.938566 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928747    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:18.938566 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928750    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:18.938566 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928753    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:18.938566 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928755    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:18.938566 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928758    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:18.938566 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928761    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:18.938566 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928763    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:18.938566 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928766    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:18.938566 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928771    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:18.938566 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928775    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:18.938566 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928778    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:18.938566 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928780    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:18.938566 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928783    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928785    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928788    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928791    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928793    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928796    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928798    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928801    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928804    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928806    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928809    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928811    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928814    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928817    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928820    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928822    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928825    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928827    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928830    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928833    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:18.938990 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928835    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928838    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928840    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928843    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928846    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928848    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928851    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928853    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928856    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928859    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928862    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928865    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928867    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928870    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928872    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928875    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928877    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928880    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928882    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928885    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:18.939478 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928888    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:18.939927 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928890    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:18.939927 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928893    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:18.939927 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928895    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:18.939927 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928898    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:18.939927 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928900    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:18.939927 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928904    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:18.939927 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928907    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:18.939927 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928909    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:18.939927 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928912    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:18.939927 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928914    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:18.939927 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928917    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:18.939927 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928919    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:18.939927 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928922    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:18.939927 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928924    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:18.939927 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:18.928927    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:18.939927 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.928976    2578 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:41:18.939927 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.932680    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 02:41:18.939927 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.933768    2578 server.go:1014] "Starting client certificate rotation"
Oct 30 02:41:18.939927 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.933867    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:41:18.940405 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.933911    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:41:18.940405 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.936211    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:41:18.940405 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.939107    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:41:18.950378 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.950355    2578 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:41:18.954426 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.954411    2578 log.go:25] "Validated CRI v1 image API"
Oct 30 02:41:18.955293 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.955266    2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:41:18.962363 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.962343    2578 fs.go:135] Filesystem UUIDs: map[0e719797-d576-4626-b058-ac75bd407120:/dev/nvme0n1p4 2bd6444e-b28d-46e0-a215-130aed9f2ded:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 30 02:41:18.962422 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.962361    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 02:41:18.968929 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.968806    2578 manager.go:217] Machine: {Timestamp:2026-10-30 02:41:18.967278472 +0000 UTC m=+0.380734596 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099363 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c48d397284ee09c574b7bc6acc833 SystemUUID:ec2c48d3-9728-4ee0-9c57-4b7bc6acc833 BootID:7d9aff31-d605-4577-82f3-83959bba5a40 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d1:85:a6:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d1:85:a6:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:a6:00:54:41:1a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:41:18.968929 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.968927    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:41:18.969040 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.969021    2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:41:18.972401 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.972370    2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:41:18.972528 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.972401    2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-241.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:41:18.972586 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.972531    2578 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:41:18.972586 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.972539    2578 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:41:18.972586 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.972569    2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:41:18.972586 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.972582    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:41:18.973789 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.973776    2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:41:18.977186 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.977162    2578 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:41:18.977359 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.977348    2578 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:41:18.978220 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.978209    2578 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:41:18.978254 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.978227    2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:41:18.978254 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.978238    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:41:18.978254 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.978246    2578 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:41:18.978383 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.978261    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:41:18.978704 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.978688    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:41:18.982155 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.982133    2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:41:18.989210 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.989193    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:41:18.990127 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.990115    2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:41:18.990173 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.990146    2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:41:18.990317 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.990305    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:41:18.990351 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.990322    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:41:18.990351 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.990329    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:41:18.990351 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.990334    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:41:18.990351 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.990341    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:41:18.990351 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.990347    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:41:18.991031 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.991023    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:41:18.991059 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.991033    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:41:18.991059 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.991040    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:41:18.991059 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.991045    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:41:18.991059 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.991054    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:41:18.991270 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.991064    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:41:18.991767 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.991757    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:41:18.991767 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.991767    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:41:18.995545 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.995529    2578 server.go:1263] "Started kubelet"
Oct 30 02:41:18.995671 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.995618    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:41:18.995671 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.995633    2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:41:18.995744 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.995689    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:41:18.996324 ip-10-0-128-241 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:41:18.997002 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.996867    2578 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:41:18.999734 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:18.999707    2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:41:19.002087 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.002070    2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 02:41:19.005154 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.005140    2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:41:19.005243 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.005140    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:41:19.005303 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.005254    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-241.ec2.internal\" not found"
Oct 30 02:41:19.005940 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.005918    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:41:19.006047 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.005955    2578 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:41:19.006047 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.005969    2578 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:41:19.006047 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.006040    2578 factory.go:55] Registering systemd factory
Oct 30 02:41:19.006217 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.006068    2578 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:41:19.006217 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.006076    2578 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:41:19.006217 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.006081    2578 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:41:19.006361 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.006317    2578 factory.go:153] Registering CRI-O factory
Oct 30 02:41:19.006361 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.006333    2578 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:41:19.006441 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.006411    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:41:19.006479 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.006443    2578 factory.go:103] Registering Raw factory
Oct 30 02:41:19.006479 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.006460    2578 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:41:19.007501 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.007485    2578 manager.go:319] Starting recovery of all containers
Oct 30 02:41:19.018978 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.018958    2578 manager.go:324] Recovery completed
Oct 30 02:41:19.023759 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.023741    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:41:19.025240 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.025223    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:41:19.025951 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.025934    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:41:19.026025 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.025970    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:41:19.026025 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.025986    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:41:19.026521 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.026508    2578 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:41:19.026521 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.026516    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:41:19.026612 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.026518    2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:41:19.026612 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.026582    2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:41:19.027263 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.027244    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:41:19.027328 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.027276    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:41:19.027328 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.027289    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:41:19.027804 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.027791    2578 policy_none.go:50] "Start"
Oct 30 02:41:19.027804 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.027806    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:41:19.027885 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.027817    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:41:19.028173 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.028157    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-241.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 02:41:19.028242 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.028164    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-241.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 02:41:19.028847 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.028835    2578 policy_none.go:44] "Start"
Oct 30 02:41:19.029353 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.028169    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-241.ec2.internal.18e32e62ea42b3e8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-241.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-241.ec2.internal,},FirstTimestamp:2026-10-30 02:41:18.995502056 +0000 UTC m=+0.408958179,LastTimestamp:2026-10-30 02:41:18.995502056 +0000 UTC m=+0.408958179,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-241.ec2.internal,}"
Oct 30 02:41:19.042378 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.042312    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-241.ec2.internal.18e32e62ec135728  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-241.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-241.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-241.ec2.internal,},FirstTimestamp:2026-10-30 02:41:19.025952552 +0000 UTC m=+0.439408681,LastTimestamp:2026-10-30 02:41:19.025952552 +0000 UTC m=+0.439408681,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-241.ec2.internal,}"
Oct 30 02:41:19.042455 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.042415    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 02:41:19.042545 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.042525    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-241.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 02:41:19.051661 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.051641    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bx86g"
Oct 30 02:41:19.062039 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.062023    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bx86g"
Oct 30 02:41:19.065208 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.065170    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 02:41:19.065275 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.065202    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-241.ec2.internal.18e32e62ec13b82b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-241.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-241.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-241.ec2.internal,},FirstTimestamp:2026-10-30 02:41:19.025977387 +0000 UTC m=+0.439433510,LastTimestamp:2026-10-30 02:41:19.025977387 +0000 UTC m=+0.439433510,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-241.ec2.internal,}"
Oct 30 02:41:19.083926 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.075747    2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:41:19.083926 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.080371    2578 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:41:19.083926 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.080392    2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:41:19.083926 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.080400    2578 server.go:87] "Starting device plugin registration server"
Oct 30 02:41:19.083926 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.080626    2578 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:41:19.083926 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.080637    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:41:19.083926 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.080757    2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:41:19.083926 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.080826    2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:41:19.083926 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.080835    2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:41:19.083926 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.081281    2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:41:19.083926 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.081333    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-241.ec2.internal\" not found"
Oct 30 02:41:19.089290 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.089265    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:41:19.090548 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.090529    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:41:19.090548 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.090552    2578 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:41:19.090686 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.090584    2578 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:41:19.090686 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.090615    2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 02:41:19.104375 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.104357    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:19.181028 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.180991    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:41:19.181756 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.181744    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:41:19.181805 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.181770    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:41:19.181805 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.181782    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:41:19.181805 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.181802    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-241.ec2.internal"
Oct 30 02:41:19.191604 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.191581    2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal"]
Oct 30 02:41:19.191652 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.191634    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:41:19.195790 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.195777    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:41:19.195835 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.195803    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:41:19.195835 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.195816    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:41:19.197289 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.197277    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:41:19.197440 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.197425    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:19.197475 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.197456    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:41:19.197976 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.197960    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:41:19.198050 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.197998    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:41:19.198050 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.198010    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:41:19.198050 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.198034    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:41:19.198050 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.198048    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:41:19.198050 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.198015    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:41:19.199335 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.199319    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:19.199383 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.199350    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:41:19.200045 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.200020    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:41:19.200115 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.200056    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:41:19.200115 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.200066    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:41:19.203001 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.202986    2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-241.ec2.internal"
Oct 30 02:41:19.203075 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.203024    2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-241.ec2.internal\": node \"ip-10-0-128-241.ec2.internal\" not found"
Oct 30 02:41:19.226813 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.226796    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-241.ec2.internal\" not found" node="ip-10-0-128-241.ec2.internal"
Oct 30 02:41:19.230923 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.230906    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-241.ec2.internal\" not found" node="ip-10-0-128-241.ec2.internal"
Oct 30 02:41:19.253751 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.253733    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-241.ec2.internal\" not found"
Oct 30 02:41:19.307114 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.307092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b7cbf3e76b96e5e2b4f68dfa17fdc26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal\" (UID: \"0b7cbf3e76b96e5e2b4f68dfa17fdc26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:19.307229 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.307119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b7cbf3e76b96e5e2b4f68dfa17fdc26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal\" (UID: \"0b7cbf3e76b96e5e2b4f68dfa17fdc26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:19.307229 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.307136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2ea871c29059b568921cda32c2dd3ba5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-241.ec2.internal\" (UID: \"2ea871c29059b568921cda32c2dd3ba5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:19.353826 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.353809    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-241.ec2.internal\" not found"
Oct 30 02:41:19.407409 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.407388    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b7cbf3e76b96e5e2b4f68dfa17fdc26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal\" (UID: \"0b7cbf3e76b96e5e2b4f68dfa17fdc26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:19.407490 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.407442    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b7cbf3e76b96e5e2b4f68dfa17fdc26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal\" (UID: \"0b7cbf3e76b96e5e2b4f68dfa17fdc26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:19.407490 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.407414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b7cbf3e76b96e5e2b4f68dfa17fdc26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal\" (UID: \"0b7cbf3e76b96e5e2b4f68dfa17fdc26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:19.407490 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.407482    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b7cbf3e76b96e5e2b4f68dfa17fdc26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal\" (UID: \"0b7cbf3e76b96e5e2b4f68dfa17fdc26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:19.407600 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.407518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2ea871c29059b568921cda32c2dd3ba5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-241.ec2.internal\" (UID: \"2ea871c29059b568921cda32c2dd3ba5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:19.407600 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.407557    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2ea871c29059b568921cda32c2dd3ba5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-241.ec2.internal\" (UID: \"2ea871c29059b568921cda32c2dd3ba5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:19.454143 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.454099    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-241.ec2.internal\" not found"
Oct 30 02:41:19.531850 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.531833    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:19.536282 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.536089    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:19.554287 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.554269    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-241.ec2.internal\" not found"
Oct 30 02:41:19.654489 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.654456    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-241.ec2.internal\" not found"
Oct 30 02:41:19.755034 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.754984    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-241.ec2.internal\" not found"
Oct 30 02:41:19.855599 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.855577    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-241.ec2.internal\" not found"
Oct 30 02:41:19.934385 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.934232    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 02:41:19.934804 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.934505    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 02:41:19.956528 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:19.956505    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-241.ec2.internal\" not found"
Oct 30 02:41:19.990700 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:19.990659    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7cbf3e76b96e5e2b4f68dfa17fdc26.slice/crio-6218f687ee9a5ad9e74be00e02ab9145085a27d8dffebf2424985e9f4f355b79 WatchSource:0}: Error finding container 6218f687ee9a5ad9e74be00e02ab9145085a27d8dffebf2424985e9f4f355b79: Status 404 returned error can't find the container with id 6218f687ee9a5ad9e74be00e02ab9145085a27d8dffebf2424985e9f4f355b79
Oct 30 02:41:19.991020 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:19.990996    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea871c29059b568921cda32c2dd3ba5.slice/crio-3f83389d86901aa144c3416cb26dd8e5d69a416640db5e895259a44226353e59 WatchSource:0}: Error finding container 3f83389d86901aa144c3416cb26dd8e5d69a416640db5e895259a44226353e59: Status 404 returned error can't find the container with id 3f83389d86901aa144c3416cb26dd8e5d69a416640db5e895259a44226353e59
Oct 30 02:41:19.996424 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:19.996405    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:41:20.057660 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:20.057613    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-241.ec2.internal\" not found"
Oct 30 02:41:20.069812 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:20.069777    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:36:19 +0000 UTC" deadline="2028-07-22 21:25:30.101905628 +0000 UTC"
Oct 30 02:41:20.069876 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:20.069812    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15162h44m10.03209828s"
Oct 30 02:41:20.079854 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:20.078071    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:20.094112 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:20.094065    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal" event={"ID":"0b7cbf3e76b96e5e2b4f68dfa17fdc26","Type":"ContainerStarted","Data":"6218f687ee9a5ad9e74be00e02ab9145085a27d8dffebf2424985e9f4f355b79"}
Oct 30 02:41:20.095049 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:20.095031    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal" event={"ID":"2ea871c29059b568921cda32c2dd3ba5","Type":"ContainerStarted","Data":"3f83389d86901aa144c3416cb26dd8e5d69a416640db5e895259a44226353e59"}
Oct 30 02:41:20.117609 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:20.117592    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:41:20.118416 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:20.118398    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:41:20.118504 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:20.118434    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:41:20.118504 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:20.118450    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:41:20.158664 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:20.158644    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-241.ec2.internal\" not found"
Oct 30 02:41:20.242623 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:20.242605    2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:20.305476 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:20.305460    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:20.339420 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:20.339377    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:41:20.341568 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:20.341551    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:20.364298 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:20.364277    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:41:20.398735 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:20.398712    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:20.928479 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:20.928449    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:20.979066 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:20.979035    2578 apiserver.go:52] "Watching apiserver"
Oct 30 02:41:21.006866 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.006835    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 02:41:21.009715 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.009692    2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:41:21.010061 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.010043    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t","openshift-dns/node-resolver-6pzrk","openshift-multus/multus-additional-cni-plugins-mjph2","openshift-multus/multus-kbn25","openshift-network-diagnostics/network-check-target-wx7bz","kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal","openshift-image-registry/node-ca-tvbs5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal","openshift-multus/network-metrics-daemon-t5747","openshift-network-operator/iptables-alerter-24lkk","kube-system/konnectivity-agent-9xpbx","openshift-cluster-node-tuning-operator/tuned-vx9zb","openshift-ovn-kubernetes/ovnkube-node-fl65k","kube-system/global-pull-secret-syncer-skc8q"]
Oct 30 02:41:21.011840 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.011814    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-24lkk"
Oct 30 02:41:21.013115 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.012977    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:21.013115 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:21.013087    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5747" podUID="60924e48-a5e3-4f2b-ab35-48479c3b730b"
Oct 30 02:41:21.014211 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.014172    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.015422 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.015402    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.017182 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.016796    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.018360 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.017991    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.020077 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.019497    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:21.020077 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:21.019606    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx7bz" podUID="d3d2494a-0ef0-4da6-b6dd-225f750bb909"
Oct 30 02:41:21.021096 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.021063    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6pzrk"
Oct 30 02:41:21.022047 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.022019    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tvbs5"
Oct 30 02:41:21.023550 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.023526    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9xpbx"
Oct 30 02:41:21.025049 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.025028    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.026451 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.026432    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:21.026547 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:21.026493    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-skc8q" podUID="8a3ebba8-b807-4256-b552-5a18ec8412cb"
Oct 30 02:41:21.033539 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.033512    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6bd55\""
Oct 30 02:41:21.034774 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.034741    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:41:21.034774 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.034744    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:41:21.034774 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.034769    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:41:21.045738 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.045113    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:41:21.045738 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.045150    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:41:21.045738 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.045206    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:41:21.045738 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.045601    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:41:21.045738 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.045685    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:41:21.046046 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.045804    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:41:21.046046 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.045842    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:41:21.046046 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.045879    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:41:21.046046 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.045969    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:41:21.046046 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.045999    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:41:21.046046 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.046036    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:41:21.046380 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.046112    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:41:21.046380 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.046134    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:41:21.046380 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.046172    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:41:21.046380 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.046187    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:41:21.046380 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.046219    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:41:21.046380 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.046279    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:41:21.046380 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.046290    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:41:21.046724 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.046390    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:41:21.047990 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.047962    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:41:21.047990 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.047979    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:41:21.065641 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.065613    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:41:21.065743 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.065690    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-95sqm"
Oct 30 02:41:21.074309 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.074288    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-95sqm"
Oct 30 02:41:21.108067 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.108044    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:41:21.117378 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117357    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a3ebba8-b807-4256-b552-5a18ec8412cb-kubelet-config\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:21.117498 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-run-netns\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.117498 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-etc-kubernetes\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.117498 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117442    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-slash\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.117498 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-cni-bin\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.117498 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpt6g\" (UniqueName: \"kubernetes.io/projected/75211257-e617-40de-bf22-878d736448d1-kube-api-access-lpt6g\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.117498 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-host\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.117813 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-tuned\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.117813 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117528    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-cni-dir\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.117813 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-run-openvswitch\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.117813 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117566    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-node-log\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.117813 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117599    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkwzt\" (UniqueName: \"kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt\") pod \"network-check-target-wx7bz\" (UID: \"d3d2494a-0ef0-4da6-b6dd-225f750bb909\") " pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:21.117813 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117619    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/38e84dd1-8ee0-4a74-8802-3e3200c96642-hosts-file\") pod \"node-resolver-6pzrk\" (UID: \"38e84dd1-8ee0-4a74-8802-3e3200c96642\") " pod="openshift-dns/node-resolver-6pzrk"
Oct 30 02:41:21.117813 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.117813 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49a36e47-9ab6-49f4-bde8-85b78ba7c208-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.117813 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-var-lib-openvswitch\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.117813 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49a36e47-9ab6-49f4-bde8-85b78ba7c208-system-cni-dir\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.117813 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpf67\" (UniqueName: \"kubernetes.io/projected/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7-kube-api-access-lpf67\") pod \"node-ca-tvbs5\" (UID: \"a2796ed5-6fd6-4231-9847-1bee8c9f9dd7\") " pod="openshift-image-registry/node-ca-tvbs5"
Oct 30 02:41:21.117813 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-hostroot\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.117813 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117733    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-log-socket\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.117813 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-env-overrides\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.117813 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117780    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl9cq\" (UniqueName: \"kubernetes.io/projected/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-kube-api-access-sl9cq\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.117813 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cshhd\" (UniqueName: \"kubernetes.io/projected/38e84dd1-8ee0-4a74-8802-3e3200c96642-kube-api-access-cshhd\") pod \"node-resolver-6pzrk\" (UID: \"38e84dd1-8ee0-4a74-8802-3e3200c96642\") " pod="openshift-dns/node-resolver-6pzrk"
Oct 30 02:41:21.118639 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-var-lib-kubelet\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.118639 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-ovnkube-script-lib\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.118639 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/895d8ed4-6421-4018-a630-fb995aadf0bc-host-slash\") pod \"iptables-alerter-24lkk\" (UID: \"895d8ed4-6421-4018-a630-fb995aadf0bc\") " pod="openshift-network-operator/iptables-alerter-24lkk"
Oct 30 02:41:21.118639 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-sys-fs\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.118639 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-sysconfig\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.118639 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-systemd\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.118639 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117960    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49a36e47-9ab6-49f4-bde8-85b78ba7c208-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.118639 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.117984    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-socket-dir-parent\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.118639 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-run-multus-certs\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.118639 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118043    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-cni-netd\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.118639 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a3ebba8-b807-4256-b552-5a18ec8412cb-dbus\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:21.118639 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-conf-dir\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.118639 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-daemon-config\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.118639 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-etc-openvswitch\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.118639 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-ovn-node-metrics-cert\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.118639 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-cnibin\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.119497 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-kubernetes\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.119497 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca9b24b8-5841-4f54-aa38-518bfbb701c6-tmp\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.119497 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49a36e47-9ab6-49f4-bde8-85b78ba7c208-cnibin\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.119497 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-ovnkube-config\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.119497 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7-serviceca\") pod \"node-ca-tvbs5\" (UID: \"a2796ed5-6fd6-4231-9847-1bee8c9f9dd7\") " pod="openshift-image-registry/node-ca-tvbs5"
Oct 30 02:41:21.119497 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118288    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.119497 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-sys\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.119497 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:21.119497 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnqw2\" (UniqueName: \"kubernetes.io/projected/56c4a0de-1d41-4b83-9bac-ddee3f43879b-kube-api-access-jnqw2\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.119497 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kh57\" (UniqueName: \"kubernetes.io/projected/895d8ed4-6421-4018-a630-fb995aadf0bc-kube-api-access-6kh57\") pod \"iptables-alerter-24lkk\" (UID: \"895d8ed4-6421-4018-a630-fb995aadf0bc\") " pod="openshift-network-operator/iptables-alerter-24lkk"
Oct 30 02:41:21.119497 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118603    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:21.119497 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-socket-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.119497 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-registration-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.119497 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-kubelet\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.119497 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118701    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-run-systemd\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.119497 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d28e48a8-f70c-4949-b407-4c0e632f6a35-konnectivity-ca\") pod \"konnectivity-agent-9xpbx\" (UID: \"d28e48a8-f70c-4949-b407-4c0e632f6a35\") " pod="kube-system/konnectivity-agent-9xpbx"
Oct 30 02:41:21.120397 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7-host\") pod \"node-ca-tvbs5\" (UID: \"a2796ed5-6fd6-4231-9847-1bee8c9f9dd7\") " pod="openshift-image-registry/node-ca-tvbs5"
Oct 30 02:41:21.120397 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-device-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.120397 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-sysctl-d\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.120397 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118795    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgfgs\" (UniqueName: \"kubernetes.io/projected/ca9b24b8-5841-4f54-aa38-518bfbb701c6-kube-api-access-vgfgs\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.120397 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sv8m\" (UniqueName: \"kubernetes.io/projected/49a36e47-9ab6-49f4-bde8-85b78ba7c208-kube-api-access-6sv8m\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.120397 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-system-cni-dir\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.120397 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118875    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e84dd1-8ee0-4a74-8802-3e3200c96642-tmp-dir\") pod \"node-resolver-6pzrk\" (UID: \"38e84dd1-8ee0-4a74-8802-3e3200c96642\") " pod="openshift-dns/node-resolver-6pzrk"
Oct 30 02:41:21.120397 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118891    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-run\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.120397 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118910    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56c4a0de-1d41-4b83-9bac-ddee3f43879b-cni-binary-copy\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.120397 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-run-k8s-cni-cncf-io\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.120397 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-run-netns\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.120397 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-run-ovn\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.120397 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118973    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-sysctl-conf\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.120397 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.118993    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-lib-modules\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.120397 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.119012    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-var-lib-kubelet\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.120397 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.119029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49a36e47-9ab6-49f4-bde8-85b78ba7c208-cni-binary-copy\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.121248 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.119045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-var-lib-cni-multus\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.121248 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.119061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-systemd-units\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.121248 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.119077    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.121248 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.119110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.121248 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.119128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/895d8ed4-6421-4018-a630-fb995aadf0bc-iptables-alerter-script\") pod \"iptables-alerter-24lkk\" (UID: \"895d8ed4-6421-4018-a630-fb995aadf0bc\") " pod="openshift-network-operator/iptables-alerter-24lkk"
Oct 30 02:41:21.121248 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.119206    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49a36e47-9ab6-49f4-bde8-85b78ba7c208-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.121248 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.119238    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-var-lib-cni-bin\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.121248 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.119309    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d28e48a8-f70c-4949-b407-4c0e632f6a35-agent-certs\") pod \"konnectivity-agent-9xpbx\" (UID: \"d28e48a8-f70c-4949-b407-4c0e632f6a35\") " pod="kube-system/konnectivity-agent-9xpbx"
Oct 30 02:41:21.121248 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.119356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqbmt\" (UniqueName: \"kubernetes.io/projected/60924e48-a5e3-4f2b-ab35-48479c3b730b-kube-api-access-lqbmt\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:21.121248 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.119393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-modprobe-d\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.142954 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.142921    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:41:21.143302 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.143072    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:41:21.143302 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.143246    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hjgzr\""
Oct 30 02:41:21.143302 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.143259    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zm8vf\""
Oct 30 02:41:21.143302 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.143281    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-29rqz\""
Oct 30 02:41:21.143570 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.143357    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2wtbz\""
Oct 30 02:41:21.143570 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.143523    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-stwqd\""
Oct 30 02:41:21.143678 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.143631    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:41:21.143737 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.143688    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zz8vj\""
Oct 30 02:41:21.143792 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.143748    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vzqmp\""
Oct 30 02:41:21.143834 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.143801    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-kxfcp\""
Oct 30 02:41:21.219738 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.219652    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.219906 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.219726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49a36e47-9ab6-49f4-bde8-85b78ba7c208-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.219906 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.219770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-var-lib-openvswitch\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.219906 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.219804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49a36e47-9ab6-49f4-bde8-85b78ba7c208-system-cni-dir\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.219906 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.219832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpf67\" (UniqueName: \"kubernetes.io/projected/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7-kube-api-access-lpf67\") pod \"node-ca-tvbs5\" (UID: \"a2796ed5-6fd6-4231-9847-1bee8c9f9dd7\") " pod="openshift-image-registry/node-ca-tvbs5"
Oct 30 02:41:21.219906 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.219866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-hostroot\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.219906 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.219895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-log-socket\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.220272 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.219929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-env-overrides\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.220272 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.219955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl9cq\" (UniqueName: \"kubernetes.io/projected/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-kube-api-access-sl9cq\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.220272 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.219985    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cshhd\" (UniqueName: \"kubernetes.io/projected/38e84dd1-8ee0-4a74-8802-3e3200c96642-kube-api-access-cshhd\") pod \"node-resolver-6pzrk\" (UID: \"38e84dd1-8ee0-4a74-8802-3e3200c96642\") " pod="openshift-dns/node-resolver-6pzrk"
Oct 30 02:41:21.220272 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-var-lib-kubelet\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.220272 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220040    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-ovnkube-script-lib\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.220272 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/895d8ed4-6421-4018-a630-fb995aadf0bc-host-slash\") pod \"iptables-alerter-24lkk\" (UID: \"895d8ed4-6421-4018-a630-fb995aadf0bc\") " pod="openshift-network-operator/iptables-alerter-24lkk"
Oct 30 02:41:21.220272 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-sys-fs\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.220272 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-sysconfig\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.220272 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-systemd\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.220272 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49a36e47-9ab6-49f4-bde8-85b78ba7c208-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.220272 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-socket-dir-parent\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.220272 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-run-multus-certs\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.220272 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220281    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-cni-netd\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.220970 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220310    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a3ebba8-b807-4256-b552-5a18ec8412cb-dbus\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:21.220970 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-conf-dir\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.220970 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-daemon-config\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.220970 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-etc-openvswitch\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.220970 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-ovn-node-metrics-cert\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.220970 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220433    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-cnibin\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.220970 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220453    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-kubernetes\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.220970 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca9b24b8-5841-4f54-aa38-518bfbb701c6-tmp\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.220970 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49a36e47-9ab6-49f4-bde8-85b78ba7c208-cnibin\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.220970 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-ovnkube-config\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.220970 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7-serviceca\") pod \"node-ca-tvbs5\" (UID: \"a2796ed5-6fd6-4231-9847-1bee8c9f9dd7\") " pod="openshift-image-registry/node-ca-tvbs5"
Oct 30 02:41:21.220970 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.220970 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220554    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-sys\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.220970 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220572    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:21.220970 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnqw2\" (UniqueName: \"kubernetes.io/projected/56c4a0de-1d41-4b83-9bac-ddee3f43879b-kube-api-access-jnqw2\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.220970 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kh57\" (UniqueName: \"kubernetes.io/projected/895d8ed4-6421-4018-a630-fb995aadf0bc-kube-api-access-6kh57\") pod \"iptables-alerter-24lkk\" (UID: \"895d8ed4-6421-4018-a630-fb995aadf0bc\") " pod="openshift-network-operator/iptables-alerter-24lkk"
Oct 30 02:41:21.220970 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:21.221881 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-socket-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.221881 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-registration-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.221881 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-kubelet\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.221881 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220711    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-run-systemd\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.221881 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d28e48a8-f70c-4949-b407-4c0e632f6a35-konnectivity-ca\") pod \"konnectivity-agent-9xpbx\" (UID: \"d28e48a8-f70c-4949-b407-4c0e632f6a35\") " pod="kube-system/konnectivity-agent-9xpbx"
Oct 30 02:41:21.221881 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7-host\") pod \"node-ca-tvbs5\" (UID: \"a2796ed5-6fd6-4231-9847-1bee8c9f9dd7\") " pod="openshift-image-registry/node-ca-tvbs5"
Oct 30 02:41:21.221881 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220763    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-device-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.221881 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-sysctl-d\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.221881 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgfgs\" (UniqueName: \"kubernetes.io/projected/ca9b24b8-5841-4f54-aa38-518bfbb701c6-kube-api-access-vgfgs\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.221881 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sv8m\" (UniqueName: \"kubernetes.io/projected/49a36e47-9ab6-49f4-bde8-85b78ba7c208-kube-api-access-6sv8m\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.221881 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-system-cni-dir\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.221881 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e84dd1-8ee0-4a74-8802-3e3200c96642-tmp-dir\") pod \"node-resolver-6pzrk\" (UID: \"38e84dd1-8ee0-4a74-8802-3e3200c96642\") " pod="openshift-dns/node-resolver-6pzrk"
Oct 30 02:41:21.221881 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-run\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.221881 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56c4a0de-1d41-4b83-9bac-ddee3f43879b-cni-binary-copy\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.221881 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-run-k8s-cni-cncf-io\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.221881 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-run-netns\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.221881 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-run-ovn\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.222808 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-sysctl-conf\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.222808 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.220999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-lib-modules\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.222808 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.221016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-var-lib-kubelet\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.222808 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.221032    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49a36e47-9ab6-49f4-bde8-85b78ba7c208-cni-binary-copy\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.222808 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.221049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-var-lib-cni-multus\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.222808 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.221064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-systemd-units\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.222808 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.221125    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-systemd-units\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.222808 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.221133    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49a36e47-9ab6-49f4-bde8-85b78ba7c208-cnibin\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.222808 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.221235    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.222808 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.221529    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7-serviceca\") pod \"node-ca-tvbs5\" (UID: \"a2796ed5-6fd6-4231-9847-1bee8c9f9dd7\") " pod="openshift-image-registry/node-ca-tvbs5"
Oct 30 02:41:21.222808 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.221931    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.222808 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.221937    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-var-lib-openvswitch\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.222808 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.221974    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-sys\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.222808 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.221984    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49a36e47-9ab6-49f4-bde8-85b78ba7c208-system-cni-dir\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.222808 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:21.222058    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:21.222808 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:21.222120    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret podName:8a3ebba8-b807-4256-b552-5a18ec8412cb nodeName:}" failed. No retries permitted until 2026-10-30 02:41:21.722100088 +0000 UTC m=+3.135556214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret") pod "global-pull-secret-syncer-skc8q" (UID: "8a3ebba8-b807-4256-b552-5a18ec8412cb") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:21.222808 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.222233    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-hostroot\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.223740 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.222298    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-log-socket\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.223740 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:21.222352    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:21.223740 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:21.222388    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs podName:60924e48-a5e3-4f2b-ab35-48479c3b730b nodeName:}" failed. No retries permitted until 2026-10-30 02:41:21.722376828 +0000 UTC m=+3.135832940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs") pod "network-metrics-daemon-t5747" (UID: "60924e48-a5e3-4f2b-ab35-48479c3b730b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:21.223740 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.222589    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49a36e47-9ab6-49f4-bde8-85b78ba7c208-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.223740 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.222655    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-socket-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.223740 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.222733    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-registration-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.223740 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.222780    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-kubelet\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.223740 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.222821    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-ovnkube-config\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.223740 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.222827    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-run-systemd\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.223740 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.222877    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-var-lib-kubelet\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.223740 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.222927    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-cni-netd\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.223740 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.222939    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/895d8ed4-6421-4018-a630-fb995aadf0bc-host-slash\") pod \"iptables-alerter-24lkk\" (UID: \"895d8ed4-6421-4018-a630-fb995aadf0bc\") " pod="openshift-network-operator/iptables-alerter-24lkk"
Oct 30 02:41:21.223740 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223002    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-sys-fs\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.223740 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223045    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-run-k8s-cni-cncf-io\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.223740 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223062    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-sysconfig\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.223740 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223082    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7-host\") pod \"node-ca-tvbs5\" (UID: \"a2796ed5-6fd6-4231-9847-1bee8c9f9dd7\") " pod="openshift-image-registry/node-ca-tvbs5"
Oct 30 02:41:21.223740 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223121    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-device-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.224703 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223121    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-systemd\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.224703 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223175    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-system-cni-dir\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.224703 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223388    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-ovnkube-script-lib\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.224703 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223451    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e84dd1-8ee0-4a74-8802-3e3200c96642-tmp-dir\") pod \"node-resolver-6pzrk\" (UID: \"38e84dd1-8ee0-4a74-8802-3e3200c96642\") " pod="openshift-dns/node-resolver-6pzrk"
Oct 30 02:41:21.224703 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223512    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d28e48a8-f70c-4949-b407-4c0e632f6a35-konnectivity-ca\") pod \"konnectivity-agent-9xpbx\" (UID: \"d28e48a8-f70c-4949-b407-4c0e632f6a35\") " pod="kube-system/konnectivity-agent-9xpbx"
Oct 30 02:41:21.224703 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.224703 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223561    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-kubernetes\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.224703 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.224703 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/895d8ed4-6421-4018-a630-fb995aadf0bc-iptables-alerter-script\") pod \"iptables-alerter-24lkk\" (UID: \"895d8ed4-6421-4018-a630-fb995aadf0bc\") " pod="openshift-network-operator/iptables-alerter-24lkk"
Oct 30 02:41:21.224703 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49a36e47-9ab6-49f4-bde8-85b78ba7c208-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.224703 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223636    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-lib-modules\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.224703 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.222870    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-env-overrides\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.224703 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223647    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-var-lib-cni-bin\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.224703 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d28e48a8-f70c-4949-b407-4c0e632f6a35-agent-certs\") pod \"konnectivity-agent-9xpbx\" (UID: \"d28e48a8-f70c-4949-b407-4c0e632f6a35\") " pod="kube-system/konnectivity-agent-9xpbx"
Oct 30 02:41:21.224703 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223670    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-run\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.224703 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqbmt\" (UniqueName: \"kubernetes.io/projected/60924e48-a5e3-4f2b-ab35-48479c3b730b-kube-api-access-lqbmt\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:21.224703 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223710    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-modprobe-d\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.225558 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a3ebba8-b807-4256-b552-5a18ec8412cb-kubelet-config\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:21.225558 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-run-netns\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.225558 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-etc-kubernetes\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.225558 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-slash\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.225558 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-cni-bin\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.225558 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpt6g\" (UniqueName: \"kubernetes.io/projected/75211257-e617-40de-bf22-878d736448d1-kube-api-access-lpt6g\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.225558 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-host\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.225558 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223880    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-tuned\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.225558 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-cni-dir\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.225558 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-run-openvswitch\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.225558 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-node-log\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.225558 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223928    2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:41:21.225558 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223947    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwzt\" (UniqueName: \"kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt\") pod \"network-check-target-wx7bz\" (UID: \"d3d2494a-0ef0-4da6-b6dd-225f750bb909\") " pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:21.225558 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/38e84dd1-8ee0-4a74-8802-3e3200c96642-hosts-file\") pod \"node-resolver-6pzrk\" (UID: \"38e84dd1-8ee0-4a74-8802-3e3200c96642\") " pod="openshift-dns/node-resolver-6pzrk"
Oct 30 02:41:21.225558 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224036    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49a36e47-9ab6-49f4-bde8-85b78ba7c208-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.225558 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224046    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49a36e47-9ab6-49f4-bde8-85b78ba7c208-cni-binary-copy\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.225558 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224043    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/38e84dd1-8ee0-4a74-8802-3e3200c96642-hosts-file\") pod \"node-resolver-6pzrk\" (UID: \"38e84dd1-8ee0-4a74-8802-3e3200c96642\") " pod="openshift-dns/node-resolver-6pzrk"
Oct 30 02:41:21.225558 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224061    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-run-netns\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.226493 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224093    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-var-lib-kubelet\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.226493 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224165    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-var-lib-cni-multus\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.226493 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224190    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-run-multus-certs\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.226493 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224291    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-sysctl-d\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.226493 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224316    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-slash\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.226493 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224315    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-etc-kubernetes\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.226493 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224362    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-socket-dir-parent\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.226493 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224474    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-host\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.226493 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224600    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-cni-bin\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.226493 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224648    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-run-netns\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.226493 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224688    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-run-ovn\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.226493 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224689    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56c4a0de-1d41-4b83-9bac-ddee3f43879b-cni-binary-copy\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.226493 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.223520    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a3ebba8-b807-4256-b552-5a18ec8412cb-dbus\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:21.226493 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224752    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-cnibin\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.226493 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224763    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-run-openvswitch\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.226493 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224796    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-conf-dir\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.226493 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224804    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-sysctl-conf\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.226493 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224879    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-modprobe-d\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.227462 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224883    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-cni-dir\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.227462 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.224922    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a3ebba8-b807-4256-b552-5a18ec8412cb-kubelet-config\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:21.227462 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.225015    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-node-log\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.227462 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.225093    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-etc-openvswitch\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.227462 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.225321    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.227462 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.225364    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.227462 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.225413    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-var-lib-cni-bin\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.227462 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.225872    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-daemon-config\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.227462 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.226042    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/895d8ed4-6421-4018-a630-fb995aadf0bc-iptables-alerter-script\") pod \"iptables-alerter-24lkk\" (UID: \"895d8ed4-6421-4018-a630-fb995aadf0bc\") " pod="openshift-network-operator/iptables-alerter-24lkk"
Oct 30 02:41:21.227462 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.226194    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49a36e47-9ab6-49f4-bde8-85b78ba7c208-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.228156 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.228130    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-tuned\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.228488 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.228466    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d28e48a8-f70c-4949-b407-4c0e632f6a35-agent-certs\") pod \"konnectivity-agent-9xpbx\" (UID: \"d28e48a8-f70c-4949-b407-4c0e632f6a35\") " pod="kube-system/konnectivity-agent-9xpbx"
Oct 30 02:41:21.228641 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.228618    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-ovn-node-metrics-cert\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.230016 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.229990    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca9b24b8-5841-4f54-aa38-518bfbb701c6-tmp\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.251033 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.251008    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnqw2\" (UniqueName: \"kubernetes.io/projected/56c4a0de-1d41-4b83-9bac-ddee3f43879b-kube-api-access-jnqw2\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.252825 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.252796    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cshhd\" (UniqueName: \"kubernetes.io/projected/38e84dd1-8ee0-4a74-8802-3e3200c96642-kube-api-access-cshhd\") pod \"node-resolver-6pzrk\" (UID: \"38e84dd1-8ee0-4a74-8802-3e3200c96642\") " pod="openshift-dns/node-resolver-6pzrk"
Oct 30 02:41:21.253054 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.252933    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpf67\" (UniqueName: \"kubernetes.io/projected/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7-kube-api-access-lpf67\") pod \"node-ca-tvbs5\" (UID: \"a2796ed5-6fd6-4231-9847-1bee8c9f9dd7\") " pod="openshift-image-registry/node-ca-tvbs5"
Oct 30 02:41:21.255204 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:21.255182    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:41:21.255317 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:21.255210    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:41:21.255317 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:21.255226    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mkwzt for pod openshift-network-diagnostics/network-check-target-wx7bz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:21.255317 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:21.255301    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt podName:d3d2494a-0ef0-4da6-b6dd-225f750bb909 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:21.7552822 +0000 UTC m=+3.168738547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mkwzt" (UniqueName: "kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt") pod "network-check-target-wx7bz" (UID: "d3d2494a-0ef0-4da6-b6dd-225f750bb909") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:21.256368 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.256341    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sv8m\" (UniqueName: \"kubernetes.io/projected/49a36e47-9ab6-49f4-bde8-85b78ba7c208-kube-api-access-6sv8m\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.257300 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.257275    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpt6g\" (UniqueName: \"kubernetes.io/projected/75211257-e617-40de-bf22-878d736448d1-kube-api-access-lpt6g\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.257816 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.257798    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgfgs\" (UniqueName: \"kubernetes.io/projected/ca9b24b8-5841-4f54-aa38-518bfbb701c6-kube-api-access-vgfgs\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.258371 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.258352    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqbmt\" (UniqueName: \"kubernetes.io/projected/60924e48-a5e3-4f2b-ab35-48479c3b730b-kube-api-access-lqbmt\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:21.259354 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.259240    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl9cq\" (UniqueName: \"kubernetes.io/projected/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-kube-api-access-sl9cq\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.261413 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.261394    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kh57\" (UniqueName: \"kubernetes.io/projected/895d8ed4-6421-4018-a630-fb995aadf0bc-kube-api-access-6kh57\") pod \"iptables-alerter-24lkk\" (UID: \"895d8ed4-6421-4018-a630-fb995aadf0bc\") " pod="openshift-network-operator/iptables-alerter-24lkk"
Oct 30 02:41:21.326207 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.326177    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-24lkk"
Oct 30 02:41:21.345272 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.345235    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:41:21.348918 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.348892    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kbn25"
Oct 30 02:41:21.356855 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.356833    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:41:21.357088 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:21.357064    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a36e47_9ab6_49f4_bde8_85b78ba7c208.slice/crio-d2fb4e9ea19f875c664825b0926b6a616905fc4f12ef6853c726a5394985bf8e WatchSource:0}: Error finding container d2fb4e9ea19f875c664825b0926b6a616905fc4f12ef6853c726a5394985bf8e: Status 404 returned error can't find the container with id d2fb4e9ea19f875c664825b0926b6a616905fc4f12ef6853c726a5394985bf8e
Oct 30 02:41:21.361763 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:21.361738    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c4a0de_1d41_4b83_9bac_ddee3f43879b.slice/crio-f28ccef2af82f377429f80311d72cbb000f56833b5b5376c0eda08d7be2fc126 WatchSource:0}: Error finding container f28ccef2af82f377429f80311d72cbb000f56833b5b5376c0eda08d7be2fc126: Status 404 returned error can't find the container with id f28ccef2af82f377429f80311d72cbb000f56833b5b5376c0eda08d7be2fc126
Oct 30 02:41:21.368603 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:21.368577    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75211257_e617_40de_bf22_878d736448d1.slice/crio-438045a0b01f1568bab7043e3ba8e47cef5fb051e72d0adb845357919c6dbd15 WatchSource:0}: Error finding container 438045a0b01f1568bab7043e3ba8e47cef5fb051e72d0adb845357919c6dbd15: Status 404 returned error can't find the container with id 438045a0b01f1568bab7043e3ba8e47cef5fb051e72d0adb845357919c6dbd15
Oct 30 02:41:21.370249 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.370184    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:21.373900 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.373738    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6pzrk"
Oct 30 02:41:21.379904 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:21.379850    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c022b3d_db3f_45a7_b6b7_5404c6f170c8.slice/crio-639d650dbcaa5a892eaaf854b9eb42ddb3692ca7e6d6e99b7ce5b9e7e9954633 WatchSource:0}: Error finding container 639d650dbcaa5a892eaaf854b9eb42ddb3692ca7e6d6e99b7ce5b9e7e9954633: Status 404 returned error can't find the container with id 639d650dbcaa5a892eaaf854b9eb42ddb3692ca7e6d6e99b7ce5b9e7e9954633
Oct 30 02:41:21.380801 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.380781    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tvbs5"
Oct 30 02:41:21.390801 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.390744    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9xpbx"
Oct 30 02:41:21.391916 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:21.391890    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e84dd1_8ee0_4a74_8802_3e3200c96642.slice/crio-ecdf5dc72cb424d2f4b3cd5ed9fd3beea31f652ce5a66a5f2954ba9dfded6365 WatchSource:0}: Error finding container ecdf5dc72cb424d2f4b3cd5ed9fd3beea31f652ce5a66a5f2954ba9dfded6365: Status 404 returned error can't find the container with id ecdf5dc72cb424d2f4b3cd5ed9fd3beea31f652ce5a66a5f2954ba9dfded6365
Oct 30 02:41:21.395326 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:21.395166    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2796ed5_6fd6_4231_9847_1bee8c9f9dd7.slice/crio-4d98adb609a438f29cc18c1911ff35f2e0c45c42c5078807bb7927506095c6c3 WatchSource:0}: Error finding container 4d98adb609a438f29cc18c1911ff35f2e0c45c42c5078807bb7927506095c6c3: Status 404 returned error can't find the container with id 4d98adb609a438f29cc18c1911ff35f2e0c45c42c5078807bb7927506095c6c3
Oct 30 02:41:21.397617 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.397566    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:41:21.401278 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:21.401254    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28e48a8_f70c_4949_b407_4c0e632f6a35.slice/crio-67b82271ccb3ea645450dcfdbe744c68725094663336981be1d336bf4de228c8 WatchSource:0}: Error finding container 67b82271ccb3ea645450dcfdbe744c68725094663336981be1d336bf4de228c8: Status 404 returned error can't find the container with id 67b82271ccb3ea645450dcfdbe744c68725094663336981be1d336bf4de228c8
Oct 30 02:41:21.407637 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:21.407615    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9b24b8_5841_4f54_aa38_518bfbb701c6.slice/crio-5f1e78abc953a7b7db5d2dcaf228d08f18dcdc25fa8267d0398d1e03df730a92 WatchSource:0}: Error finding container 5f1e78abc953a7b7db5d2dcaf228d08f18dcdc25fa8267d0398d1e03df730a92: Status 404 returned error can't find the container with id 5f1e78abc953a7b7db5d2dcaf228d08f18dcdc25fa8267d0398d1e03df730a92
Oct 30 02:41:21.727996 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.727952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:21.727996 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.728005    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:21.728188 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:21.728081    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:21.728188 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:21.728090    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:21.728188 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:21.728140    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret podName:8a3ebba8-b807-4256-b552-5a18ec8412cb nodeName:}" failed. No retries permitted until 2026-10-30 02:41:22.72812551 +0000 UTC m=+4.141581639 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret") pod "global-pull-secret-syncer-skc8q" (UID: "8a3ebba8-b807-4256-b552-5a18ec8412cb") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:21.728188 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:21.728156    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs podName:60924e48-a5e3-4f2b-ab35-48479c3b730b nodeName:}" failed. No retries permitted until 2026-10-30 02:41:22.728148645 +0000 UTC m=+4.141604756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs") pod "network-metrics-daemon-t5747" (UID: "60924e48-a5e3-4f2b-ab35-48479c3b730b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:21.829160 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:21.829073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwzt\" (UniqueName: \"kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt\") pod \"network-check-target-wx7bz\" (UID: \"d3d2494a-0ef0-4da6-b6dd-225f750bb909\") " pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:21.829337 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:21.829252    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:41:21.829337 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:21.829277    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:41:21.829337 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:21.829290    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mkwzt for pod openshift-network-diagnostics/network-check-target-wx7bz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:21.829515 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:21.829361    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt podName:d3d2494a-0ef0-4da6-b6dd-225f750bb909 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:22.829342475 +0000 UTC m=+4.242798657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mkwzt" (UniqueName: "kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt") pod "network-check-target-wx7bz" (UID: "d3d2494a-0ef0-4da6-b6dd-225f750bb909") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:22.077405 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:22.077340    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:36:21 +0000 UTC" deadline="2028-06-28 18:23:56.974359927 +0000 UTC"
Oct 30 02:41:22.077405 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:22.077404    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14583h42m34.896960922s"
Oct 30 02:41:22.125064 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:22.124977    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjph2" event={"ID":"49a36e47-9ab6-49f4-bde8-85b78ba7c208","Type":"ContainerStarted","Data":"d2fb4e9ea19f875c664825b0926b6a616905fc4f12ef6853c726a5394985bf8e"}
Oct 30 02:41:22.129773 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:22.129746    2578 generic.go:356] "Generic (PLEG): container finished" podID="0b7cbf3e76b96e5e2b4f68dfa17fdc26" containerID="2896b4b72c34dd2d3a9ecc96dec656d0c7168fa718181d889718debb257591a1" exitCode=0
Oct 30 02:41:22.129890 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:22.129818    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal" event={"ID":"0b7cbf3e76b96e5e2b4f68dfa17fdc26","Type":"ContainerDied","Data":"2896b4b72c34dd2d3a9ecc96dec656d0c7168fa718181d889718debb257591a1"}
Oct 30 02:41:22.138023 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:22.137995    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tvbs5" event={"ID":"a2796ed5-6fd6-4231-9847-1bee8c9f9dd7","Type":"ContainerStarted","Data":"4d98adb609a438f29cc18c1911ff35f2e0c45c42c5078807bb7927506095c6c3"}
Oct 30 02:41:22.143184 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:22.143155    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-24lkk" event={"ID":"895d8ed4-6421-4018-a630-fb995aadf0bc","Type":"ContainerStarted","Data":"32a22c148d94150ee483058da90f16b3b35fbf12fc9824acf14aa476872056c4"}
Oct 30 02:41:22.158017 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:22.157983    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal" event={"ID":"2ea871c29059b568921cda32c2dd3ba5","Type":"ContainerStarted","Data":"7c92bf6543ded960e6603ecfd73d56610e94917dd22fc50dcd19131eb3a2e417"}
Oct 30 02:41:22.170347 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:22.170321    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vx9zb" event={"ID":"ca9b24b8-5841-4f54-aa38-518bfbb701c6","Type":"ContainerStarted","Data":"5f1e78abc953a7b7db5d2dcaf228d08f18dcdc25fa8267d0398d1e03df730a92"}
Oct 30 02:41:22.174627 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:22.174601    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9xpbx" event={"ID":"d28e48a8-f70c-4949-b407-4c0e632f6a35","Type":"ContainerStarted","Data":"67b82271ccb3ea645450dcfdbe744c68725094663336981be1d336bf4de228c8"}
Oct 30 02:41:22.209247 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:22.209217    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6pzrk" event={"ID":"38e84dd1-8ee0-4a74-8802-3e3200c96642","Type":"ContainerStarted","Data":"ecdf5dc72cb424d2f4b3cd5ed9fd3beea31f652ce5a66a5f2954ba9dfded6365"}
Oct 30 02:41:22.238078 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:22.238048    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k" event={"ID":"3c022b3d-db3f-45a7-b6b7-5404c6f170c8","Type":"ContainerStarted","Data":"639d650dbcaa5a892eaaf854b9eb42ddb3692ca7e6d6e99b7ce5b9e7e9954633"}
Oct 30 02:41:22.250012 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:22.249978    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t" event={"ID":"75211257-e617-40de-bf22-878d736448d1","Type":"ContainerStarted","Data":"438045a0b01f1568bab7043e3ba8e47cef5fb051e72d0adb845357919c6dbd15"}
Oct 30 02:41:22.273273 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:22.273174    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kbn25" event={"ID":"56c4a0de-1d41-4b83-9bac-ddee3f43879b","Type":"ContainerStarted","Data":"f28ccef2af82f377429f80311d72cbb000f56833b5b5376c0eda08d7be2fc126"}
Oct 30 02:41:22.738034 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:22.736769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:22.738034 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:22.737061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:22.738034 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:22.737272    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:22.738034 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:22.737342    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs podName:60924e48-a5e3-4f2b-ab35-48479c3b730b nodeName:}" failed. No retries permitted until 2026-10-30 02:41:24.737322484 +0000 UTC m=+6.150778624 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs") pod "network-metrics-daemon-t5747" (UID: "60924e48-a5e3-4f2b-ab35-48479c3b730b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:22.738034 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:22.737813    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:22.738034 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:22.737871    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret podName:8a3ebba8-b807-4256-b552-5a18ec8412cb nodeName:}" failed. No retries permitted until 2026-10-30 02:41:24.737854298 +0000 UTC m=+6.151310415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret") pod "global-pull-secret-syncer-skc8q" (UID: "8a3ebba8-b807-4256-b552-5a18ec8412cb") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:22.838599 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:22.837909    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwzt\" (UniqueName: \"kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt\") pod \"network-check-target-wx7bz\" (UID: \"d3d2494a-0ef0-4da6-b6dd-225f750bb909\") " pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:22.838599 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:22.838093    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:41:22.838599 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:22.838120    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:41:22.838599 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:22.838134    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mkwzt for pod openshift-network-diagnostics/network-check-target-wx7bz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:22.838599 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:22.838202    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt podName:d3d2494a-0ef0-4da6-b6dd-225f750bb909 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:24.838180363 +0000 UTC m=+6.251636489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mkwzt" (UniqueName: "kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt") pod "network-check-target-wx7bz" (UID: "d3d2494a-0ef0-4da6-b6dd-225f750bb909") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:23.078295 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:23.078170    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:36:21 +0000 UTC" deadline="2028-04-14 04:04:13.882627631 +0000 UTC"
Oct 30 02:41:23.078295 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:23.078235    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12769h22m50.804398327s"
Oct 30 02:41:23.091802 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:23.091767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:23.091955 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:23.091930    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5747" podUID="60924e48-a5e3-4f2b-ab35-48479c3b730b"
Oct 30 02:41:23.092406 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:23.092382    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:23.092536 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:23.092492    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-skc8q" podUID="8a3ebba8-b807-4256-b552-5a18ec8412cb"
Oct 30 02:41:23.092773 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:23.092742    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:23.092857 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:23.092833    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx7bz" podUID="d3d2494a-0ef0-4da6-b6dd-225f750bb909"
Oct 30 02:41:23.302485 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:23.302449    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal" event={"ID":"0b7cbf3e76b96e5e2b4f68dfa17fdc26","Type":"ContainerStarted","Data":"7515a82683aa7c001d2ba067d4f8d6d8b641fbe12303dbd0a533ad2cbe3f801d"}
Oct 30 02:41:23.351046 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:23.350939    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal" podStartSLOduration=3.35091858 podStartE2EDuration="3.35091858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:22.2520475 +0000 UTC m=+3.665503635" watchObservedRunningTime="2026-10-30 02:41:23.35091858 +0000 UTC m=+4.764374716"
Oct 30 02:41:24.322708 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:24.321858    2578 generic.go:356] "Generic (PLEG): container finished" podID="49a36e47-9ab6-49f4-bde8-85b78ba7c208" containerID="7b9f41d0ceb1ebf685a9930181567c76fb529f39b89765c7478c8837c69c3242" exitCode=0
Oct 30 02:41:24.322708 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:24.321990    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjph2" event={"ID":"49a36e47-9ab6-49f4-bde8-85b78ba7c208","Type":"ContainerDied","Data":"7b9f41d0ceb1ebf685a9930181567c76fb529f39b89765c7478c8837c69c3242"}
Oct 30 02:41:24.408421 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:24.407700    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal" podStartSLOduration=4.407680014 podStartE2EDuration="4.407680014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:23.352167821 +0000 UTC m=+4.765623976" watchObservedRunningTime="2026-10-30 02:41:24.407680014 +0000 UTC m=+5.821136151"
Oct 30 02:41:24.756231 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:24.755325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:24.756231 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:24.755379    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:24.756231 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:24.755544    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:24.756231 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:24.755609    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs podName:60924e48-a5e3-4f2b-ab35-48479c3b730b nodeName:}" failed. No retries permitted until 2026-10-30 02:41:28.755589473 +0000 UTC m=+10.169045603 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs") pod "network-metrics-daemon-t5747" (UID: "60924e48-a5e3-4f2b-ab35-48479c3b730b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:24.756231 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:24.756081    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:24.756231 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:24.756132    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret podName:8a3ebba8-b807-4256-b552-5a18ec8412cb nodeName:}" failed. No retries permitted until 2026-10-30 02:41:28.756115931 +0000 UTC m=+10.169572043 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret") pod "global-pull-secret-syncer-skc8q" (UID: "8a3ebba8-b807-4256-b552-5a18ec8412cb") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:24.856777 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:24.856196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwzt\" (UniqueName: \"kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt\") pod \"network-check-target-wx7bz\" (UID: \"d3d2494a-0ef0-4da6-b6dd-225f750bb909\") " pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:24.856777 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:24.856391    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:41:24.856777 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:24.856412    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:41:24.856777 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:24.856436    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mkwzt for pod openshift-network-diagnostics/network-check-target-wx7bz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:24.856777 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:24.856497    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt podName:d3d2494a-0ef0-4da6-b6dd-225f750bb909 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:28.856478168 +0000 UTC m=+10.269934282 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mkwzt" (UniqueName: "kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt") pod "network-check-target-wx7bz" (UID: "d3d2494a-0ef0-4da6-b6dd-225f750bb909") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:25.092684 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:25.091937    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:25.092684 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:25.092066    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx7bz" podUID="d3d2494a-0ef0-4da6-b6dd-225f750bb909"
Oct 30 02:41:25.092684 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:25.092483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:25.092684 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:25.092598    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5747" podUID="60924e48-a5e3-4f2b-ab35-48479c3b730b"
Oct 30 02:41:25.092684 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:25.092659    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:25.093073 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:25.092728    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-skc8q" podUID="8a3ebba8-b807-4256-b552-5a18ec8412cb"
Oct 30 02:41:26.336283 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:26.335768    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9xpbx" event={"ID":"d28e48a8-f70c-4949-b407-4c0e632f6a35","Type":"ContainerStarted","Data":"6309b0caf2fdffe939f41c67d387a577bfe95203d5fd2278a204c39005e47478"}
Oct 30 02:41:27.091677 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:27.091644    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:27.091851 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:27.091792    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-skc8q" podUID="8a3ebba8-b807-4256-b552-5a18ec8412cb"
Oct 30 02:41:27.091915 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:27.091863    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:27.091974 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:27.091937    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx7bz" podUID="d3d2494a-0ef0-4da6-b6dd-225f750bb909"
Oct 30 02:41:27.092031 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:27.091988    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:27.092090 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:27.092072    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5747" podUID="60924e48-a5e3-4f2b-ab35-48479c3b730b"
Oct 30 02:41:28.250537 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:28.250376    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9xpbx"
Oct 30 02:41:28.251372 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:28.250990    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9xpbx"
Oct 30 02:41:28.295685 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:28.295630    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9xpbx" podStartSLOduration=4.178361769 podStartE2EDuration="8.295609638s" totalImagesPullingTime="4.117247869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:26.385884007 +0000 UTC m=+7.799340143" watchObservedRunningTime="2026-10-30 02:41:28.295609638 +0000 UTC m=+9.709065776"
Oct 30 02:41:28.348379 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:28.348284    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t" event={"ID":"75211257-e617-40de-bf22-878d736448d1","Type":"ContainerStarted","Data":"d851336cd1f608283b8363ed1231b9157786928dbf2ae75813fbcec7b997e10b"}
Oct 30 02:41:28.353556 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:28.353328    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tvbs5" event={"ID":"a2796ed5-6fd6-4231-9847-1bee8c9f9dd7","Type":"ContainerStarted","Data":"b9a449135a42a5cc0e04ef75cc0af5fdd1c4902ea6d303fac3df0fc5ec1a6d34"}
Oct 30 02:41:28.353556 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:28.353375    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9xpbx"
Oct 30 02:41:28.359290 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:28.354399    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9xpbx"
Oct 30 02:41:28.399911 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:28.399647    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tvbs5" podStartSLOduration=2.684584502 podStartE2EDuration="8.399627986s" totalImagesPullingTime="5.715043484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:28.399360478 +0000 UTC m=+9.812816617" watchObservedRunningTime="2026-10-30 02:41:28.399627986 +0000 UTC m=+9.813084123"
Oct 30 02:41:28.795173 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:28.794386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:28.795173 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:28.794439    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:28.795173 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:28.794615    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:28.795173 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:28.794674    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs podName:60924e48-a5e3-4f2b-ab35-48479c3b730b nodeName:}" failed. No retries permitted until 2026-10-30 02:41:36.79465515 +0000 UTC m=+18.208111277 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs") pod "network-metrics-daemon-t5747" (UID: "60924e48-a5e3-4f2b-ab35-48479c3b730b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:28.795173 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:28.795087    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:28.795173 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:28.795136    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret podName:8a3ebba8-b807-4256-b552-5a18ec8412cb nodeName:}" failed. No retries permitted until 2026-10-30 02:41:36.795121132 +0000 UTC m=+18.208577248 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret") pod "global-pull-secret-syncer-skc8q" (UID: "8a3ebba8-b807-4256-b552-5a18ec8412cb") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:28.895714 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:28.895267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwzt\" (UniqueName: \"kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt\") pod \"network-check-target-wx7bz\" (UID: \"d3d2494a-0ef0-4da6-b6dd-225f750bb909\") " pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:28.895714 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:28.895390    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:41:28.895714 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:28.895403    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:41:28.895714 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:28.895411    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mkwzt for pod openshift-network-diagnostics/network-check-target-wx7bz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:28.895714 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:28.895454    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt podName:d3d2494a-0ef0-4da6-b6dd-225f750bb909 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:36.895442341 +0000 UTC m=+18.308898465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mkwzt" (UniqueName: "kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt") pod "network-check-target-wx7bz" (UID: "d3d2494a-0ef0-4da6-b6dd-225f750bb909") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:29.097411 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:29.094408    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:29.097411 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:29.094567    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx7bz" podUID="d3d2494a-0ef0-4da6-b6dd-225f750bb909"
Oct 30 02:41:29.097411 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:29.094949    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:29.097411 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:29.095040    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5747" podUID="60924e48-a5e3-4f2b-ab35-48479c3b730b"
Oct 30 02:41:29.097411 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:29.096034    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:29.097411 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:29.096130    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-skc8q" podUID="8a3ebba8-b807-4256-b552-5a18ec8412cb"
Oct 30 02:41:29.358323 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:29.358192    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjph2" event={"ID":"49a36e47-9ab6-49f4-bde8-85b78ba7c208","Type":"ContainerStarted","Data":"74b4bbd24ffe5a1ac85ea66f35814e899d37d84743566d1f40181fe3907cf8c7"}
Oct 30 02:41:29.454017 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:29.453992    2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:41:30.090148 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:30.089104    2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:41:29.454013451Z","UUID":"833e2b4d-e52f-48da-8f20-a7bfef6e5b79","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:41:30.098095 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:30.098061    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:41:30.098263 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:30.098113    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:41:30.363928 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:30.363084    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6pzrk" event={"ID":"38e84dd1-8ee0-4a74-8802-3e3200c96642","Type":"ContainerStarted","Data":"ecddcd58fa65832dfbdd1b777f1e7403955238779e82cc5e5cae52c19731defa"}
Oct 30 02:41:30.367636 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:30.367552    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t" event={"ID":"75211257-e617-40de-bf22-878d736448d1","Type":"ContainerStarted","Data":"a32cafe2dd784032dca219bc06f27958f8c4d9c04e29753f864be46c351152a2"}
Oct 30 02:41:30.369640 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:30.369269    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kbn25" event={"ID":"56c4a0de-1d41-4b83-9bac-ddee3f43879b","Type":"ContainerStarted","Data":"dc23ff28b40fcdf1e1209281e3a0e65763bc39725dded02aacd84e4f987210d1"}
Oct 30 02:41:30.371887 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:30.371770    2578 generic.go:356] "Generic (PLEG): container finished" podID="49a36e47-9ab6-49f4-bde8-85b78ba7c208" containerID="74b4bbd24ffe5a1ac85ea66f35814e899d37d84743566d1f40181fe3907cf8c7" exitCode=0
Oct 30 02:41:30.371887 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:30.371826    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjph2" event={"ID":"49a36e47-9ab6-49f4-bde8-85b78ba7c208","Type":"ContainerDied","Data":"74b4bbd24ffe5a1ac85ea66f35814e899d37d84743566d1f40181fe3907cf8c7"}
Oct 30 02:41:30.460219 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:30.460149    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-6pzrk" podStartSLOduration=2.390943707 podStartE2EDuration="10.46013227s" totalImagesPullingTime="8.069188563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:30.410826862 +0000 UTC m=+11.824282995" watchObservedRunningTime="2026-10-30 02:41:30.46013227 +0000 UTC m=+11.873588402"
Oct 30 02:41:30.506821 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:30.506766    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-kbn25" podStartSLOduration=2.943671266 podStartE2EDuration="11.506748487s" totalImagesPullingTime="8.563077221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:30.5057861 +0000 UTC m=+11.919242232" watchObservedRunningTime="2026-10-30 02:41:30.506748487 +0000 UTC m=+11.920204621"
Oct 30 02:41:31.094807 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:31.094590    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:31.094907 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:31.094889    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx7bz" podUID="d3d2494a-0ef0-4da6-b6dd-225f750bb909"
Oct 30 02:41:31.095239 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:31.095224    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:31.095348 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:31.095329    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5747" podUID="60924e48-a5e3-4f2b-ab35-48479c3b730b"
Oct 30 02:41:31.095437 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:31.095416    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:31.095546 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:31.095525    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-skc8q" podUID="8a3ebba8-b807-4256-b552-5a18ec8412cb"
Oct 30 02:41:31.377289 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:31.377202    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t" event={"ID":"75211257-e617-40de-bf22-878d736448d1","Type":"ContainerStarted","Data":"ee69236599357d2f9f9a970c68fc38e4e540592de83b160865d498bd4dc9028e"}
Oct 30 02:41:31.380451 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:31.380413    2578 generic.go:356] "Generic (PLEG): container finished" podID="49a36e47-9ab6-49f4-bde8-85b78ba7c208" containerID="d9fe334ef5a800fb4e897389e8b4d44ca9c131908f1b0bfa021c65491f778ae6" exitCode=0
Oct 30 02:41:31.380565 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:31.380491    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjph2" event={"ID":"49a36e47-9ab6-49f4-bde8-85b78ba7c208","Type":"ContainerDied","Data":"d9fe334ef5a800fb4e897389e8b4d44ca9c131908f1b0bfa021c65491f778ae6"}
Oct 30 02:41:31.425822 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:31.425769    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t" podStartSLOduration=3.583190481 podStartE2EDuration="12.425750962s" totalImagesPullingTime="8.842560481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:31.425431426 +0000 UTC m=+12.838887559" watchObservedRunningTime="2026-10-30 02:41:31.425750962 +0000 UTC m=+12.839207114"
Oct 30 02:41:32.384698 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:32.384613    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-24lkk" event={"ID":"895d8ed4-6421-4018-a630-fb995aadf0bc","Type":"ContainerStarted","Data":"a46ab9adb7b1e3b8bbd00ef56bd9719d20d3274b7f746f1b384f63855b013319"}
Oct 30 02:41:32.386632 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:32.386596    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vx9zb" event={"ID":"ca9b24b8-5841-4f54-aa38-518bfbb701c6","Type":"ContainerStarted","Data":"e34a22d0667f588ae17da250136d0b1b9ea95bb42a5a097f8f502aa38cf8c969"}
Oct 30 02:41:32.425907 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:32.425846    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-24lkk" podStartSLOduration=4.789139798 podStartE2EDuration="13.425830199s" totalImagesPullingTime="8.636690401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:32.425267438 +0000 UTC m=+13.838723572" watchObservedRunningTime="2026-10-30 02:41:32.425830199 +0000 UTC m=+13.839286334"
Oct 30 02:41:32.468483 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:32.468439    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vx9zb" podStartSLOduration=1.748517732 podStartE2EDuration="12.468427057s" totalImagesPullingTime="10.719909325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:32.468314156 +0000 UTC m=+13.881770289" watchObservedRunningTime="2026-10-30 02:41:32.468427057 +0000 UTC m=+13.881883188"
Oct 30 02:41:33.091582 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:33.091551    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:33.091582 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:33.091582    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:33.091827 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:33.091590    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:33.091827 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:33.091679    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-skc8q" podUID="8a3ebba8-b807-4256-b552-5a18ec8412cb"
Oct 30 02:41:33.091827 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:33.091783    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5747" podUID="60924e48-a5e3-4f2b-ab35-48479c3b730b"
Oct 30 02:41:33.091953 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:33.091842    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx7bz" podUID="d3d2494a-0ef0-4da6-b6dd-225f750bb909"
Oct 30 02:41:33.391685 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:33.391607    2578 generic.go:356] "Generic (PLEG): container finished" podID="49a36e47-9ab6-49f4-bde8-85b78ba7c208" containerID="c6788018b54c1b9f0330524652016330ef1b1881cec4a1d3f10b1c9307eae839" exitCode=0
Oct 30 02:41:33.392169 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:33.391680    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjph2" event={"ID":"49a36e47-9ab6-49f4-bde8-85b78ba7c208","Type":"ContainerDied","Data":"c6788018b54c1b9f0330524652016330ef1b1881cec4a1d3f10b1c9307eae839"}
Oct 30 02:41:35.093967 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:35.093941    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:35.094449 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:35.094007    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:35.094449 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:35.094025    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:35.094449 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:35.094144    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5747" podUID="60924e48-a5e3-4f2b-ab35-48479c3b730b"
Oct 30 02:41:35.094449 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:35.094156    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-skc8q" podUID="8a3ebba8-b807-4256-b552-5a18ec8412cb"
Oct 30 02:41:35.094449 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:35.094298    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx7bz" podUID="d3d2494a-0ef0-4da6-b6dd-225f750bb909"
Oct 30 02:41:35.399638 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:35.399562    2578 generic.go:356] "Generic (PLEG): container finished" podID="49a36e47-9ab6-49f4-bde8-85b78ba7c208" containerID="55793a574377f734b917dad5f8cf2a699314270fe3c4a8d8c69224a8da0577bd" exitCode=0
Oct 30 02:41:35.400706 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:35.400678    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjph2" event={"ID":"49a36e47-9ab6-49f4-bde8-85b78ba7c208","Type":"ContainerDied","Data":"55793a574377f734b917dad5f8cf2a699314270fe3c4a8d8c69224a8da0577bd"}
Oct 30 02:41:36.403629 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:36.403449    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 02:41:36.403983 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:36.403964    2578 generic.go:356] "Generic (PLEG): container finished" podID="3c022b3d-db3f-45a7-b6b7-5404c6f170c8" containerID="a9f99c54021d76c2a97d213bdb0ad3f0cb93d53eaf6b1fb1fee25f25e8e0556a" exitCode=1
Oct 30 02:41:36.404059 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:36.404040    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k" event={"ID":"3c022b3d-db3f-45a7-b6b7-5404c6f170c8","Type":"ContainerDied","Data":"a9f99c54021d76c2a97d213bdb0ad3f0cb93d53eaf6b1fb1fee25f25e8e0556a"}
Oct 30 02:41:36.404136 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:36.404122    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k" event={"ID":"3c022b3d-db3f-45a7-b6b7-5404c6f170c8","Type":"ContainerStarted","Data":"a6b81b5aa1b36bb7a2f678bcb8ff0d143d4789f65d59eef873c6826cf082df69"}
Oct 30 02:41:36.407531 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:36.407426    2578 generic.go:356] "Generic (PLEG): container finished" podID="49a36e47-9ab6-49f4-bde8-85b78ba7c208" containerID="8b089dca29ad4871f6d5b1917646d47654ef68c40be7b73fca8c6bf80d173f18" exitCode=0
Oct 30 02:41:36.407531 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:36.407463    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjph2" event={"ID":"49a36e47-9ab6-49f4-bde8-85b78ba7c208","Type":"ContainerDied","Data":"8b089dca29ad4871f6d5b1917646d47654ef68c40be7b73fca8c6bf80d173f18"}
Oct 30 02:41:36.857674 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:36.857415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:36.857674 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:36.857652    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:36.857822 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:36.857560    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:36.857822 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:36.857742    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:36.857822 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:36.857766    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret podName:8a3ebba8-b807-4256-b552-5a18ec8412cb nodeName:}" failed. No retries permitted until 2026-10-30 02:41:52.857746729 +0000 UTC m=+34.271202842 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret") pod "global-pull-secret-syncer-skc8q" (UID: "8a3ebba8-b807-4256-b552-5a18ec8412cb") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:36.857822 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:36.857783    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs podName:60924e48-a5e3-4f2b-ab35-48479c3b730b nodeName:}" failed. No retries permitted until 2026-10-30 02:41:52.857774317 +0000 UTC m=+34.271230429 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs") pod "network-metrics-daemon-t5747" (UID: "60924e48-a5e3-4f2b-ab35-48479c3b730b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:36.958385 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:36.958357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwzt\" (UniqueName: \"kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt\") pod \"network-check-target-wx7bz\" (UID: \"d3d2494a-0ef0-4da6-b6dd-225f750bb909\") " pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:36.958522 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:36.958506    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:41:36.958561 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:36.958528    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:41:36.958561 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:36.958537    2578 projected.go:196] Error preparing data for projected volume kube-api-access-mkwzt for pod openshift-network-diagnostics/network-check-target-wx7bz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:36.958655 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:36.958580    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt podName:d3d2494a-0ef0-4da6-b6dd-225f750bb909 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:52.958567797 +0000 UTC m=+34.372023960 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mkwzt" (UniqueName: "kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt") pod "network-check-target-wx7bz" (UID: "d3d2494a-0ef0-4da6-b6dd-225f750bb909") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:37.091321 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:37.091296    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:37.091427 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:37.091360    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:37.091427 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:37.091376    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:37.091499 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:37.091457    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx7bz" podUID="d3d2494a-0ef0-4da6-b6dd-225f750bb909"
Oct 30 02:41:37.091578 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:37.091557    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-skc8q" podUID="8a3ebba8-b807-4256-b552-5a18ec8412cb"
Oct 30 02:41:37.091650 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:37.091633    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5747" podUID="60924e48-a5e3-4f2b-ab35-48479c3b730b"
Oct 30 02:41:37.413078 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:37.413046    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjph2" event={"ID":"49a36e47-9ab6-49f4-bde8-85b78ba7c208","Type":"ContainerStarted","Data":"928c2d612c7dbaa3c863d38856f29aa9772949b7d5f640b85084fbd91bd0cc9f"}
Oct 30 02:41:37.415795 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:37.415778    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 02:41:37.416100 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:37.416082    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k" event={"ID":"3c022b3d-db3f-45a7-b6b7-5404c6f170c8","Type":"ContainerStarted","Data":"ba6d2ea50c3e3233ebcbd468692f32965ab85fef75de96b595ce8bb10a102491"}
Oct 30 02:41:37.416171 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:37.416153    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k" event={"ID":"3c022b3d-db3f-45a7-b6b7-5404c6f170c8","Type":"ContainerStarted","Data":"2749a5cbc7a9037947b9fcd025c1272781ca123192e0804d4d4335db355e8c4b"}
Oct 30 02:41:37.416256 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:37.416226    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k" event={"ID":"3c022b3d-db3f-45a7-b6b7-5404c6f170c8","Type":"ContainerStarted","Data":"85f6f9bba59a84052e5bf9a23cedfadc84f1f61d0e6402e8b5af7847ae787ec1"}
Oct 30 02:41:37.416307 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:37.416276    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k" event={"ID":"3c022b3d-db3f-45a7-b6b7-5404c6f170c8","Type":"ContainerStarted","Data":"85d380bf26285fd50b41458c93602d85c9213a3a4fa1f9b7b8b63f749e496f22"}
Oct 30 02:41:37.457350 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:37.457309    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mjph2" podStartSLOduration=7.868919451 podStartE2EDuration="18.457297077s" totalImagesPullingTime="10.588377626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:37.45669877 +0000 UTC m=+18.870154902" watchObservedRunningTime="2026-10-30 02:41:37.457297077 +0000 UTC m=+18.870753208"
Oct 30 02:41:39.093605 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:39.093534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:39.093959 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:39.093620    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-skc8q" podUID="8a3ebba8-b807-4256-b552-5a18ec8412cb"
Oct 30 02:41:39.093959 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:39.093695    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:39.093959 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:39.093766    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx7bz" podUID="d3d2494a-0ef0-4da6-b6dd-225f750bb909"
Oct 30 02:41:39.093959 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:39.093774    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:39.093959 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:39.093846    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5747" podUID="60924e48-a5e3-4f2b-ab35-48479c3b730b"
Oct 30 02:41:39.423172 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:39.423110    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 02:41:39.423463 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:39.423441    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k" event={"ID":"3c022b3d-db3f-45a7-b6b7-5404c6f170c8","Type":"ContainerStarted","Data":"472a5e2e4095f8178f312c2a486e865b7b57432af2f8c0f5a478f8e0fa725ad9"}
Oct 30 02:41:41.093691 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:41.093626    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:41.093691 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:41.093658    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:41.094320 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:41.093713    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx7bz" podUID="d3d2494a-0ef0-4da6-b6dd-225f750bb909"
Oct 30 02:41:41.094320 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:41.093751    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:41.094320 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:41.093823    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-skc8q" podUID="8a3ebba8-b807-4256-b552-5a18ec8412cb"
Oct 30 02:41:41.094320 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:41.093874    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5747" podUID="60924e48-a5e3-4f2b-ab35-48479c3b730b"
Oct 30 02:41:41.430323 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:41.430263    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 02:41:41.430615 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:41.430596    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k" event={"ID":"3c022b3d-db3f-45a7-b6b7-5404c6f170c8","Type":"ContainerStarted","Data":"b721ea925878a2975ef42b04b6d47ade6f775f1b8fed64bd897ab6512721e0ca"}
Oct 30 02:41:41.430927 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:41.430911    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:41.431013 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:41.430970    2578 scope.go:122] "RemoveContainer" containerID="a9f99c54021d76c2a97d213bdb0ad3f0cb93d53eaf6b1fb1fee25f25e8e0556a"
Oct 30 02:41:41.447519 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:41.447497    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:42.437740 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:42.437710    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 02:41:42.438445 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:42.438061    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k" event={"ID":"3c022b3d-db3f-45a7-b6b7-5404c6f170c8","Type":"ContainerStarted","Data":"5056b06283e72f7f22b1e68e9a23abfbd3968fe557b8c29bd4bad0e5d741afb3"}
Oct 30 02:41:42.438445 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:42.438139    2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:41:42.438445 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:42.438396    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:42.454994 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:42.454967    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:42.530880 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:42.530836    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k" podStartSLOduration=8.703396046 podStartE2EDuration="23.530825065s" totalImagesPullingTime="14.827429019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:42.516642728 +0000 UTC m=+23.930098860" watchObservedRunningTime="2026-10-30 02:41:42.530825065 +0000 UTC m=+23.944281197"
Oct 30 02:41:42.817139 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:42.817108    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wx7bz"]
Oct 30 02:41:42.817295 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:42.817249    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:42.817348 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:42.817328    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx7bz" podUID="d3d2494a-0ef0-4da6-b6dd-225f750bb909"
Oct 30 02:41:42.817755 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:42.817729    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-skc8q"]
Oct 30 02:41:42.817843 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:42.817829    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:42.817948 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:42.817930    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-skc8q" podUID="8a3ebba8-b807-4256-b552-5a18ec8412cb"
Oct 30 02:41:42.819854 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:42.819836    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t5747"]
Oct 30 02:41:42.819931 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:42.819920    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:42.820013 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:42.819998    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5747" podUID="60924e48-a5e3-4f2b-ab35-48479c3b730b"
Oct 30 02:41:43.440353 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:43.440207    2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:41:44.091116 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:44.091089    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:44.091323 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:44.091241    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5747" podUID="60924e48-a5e3-4f2b-ab35-48479c3b730b"
Oct 30 02:41:44.442633 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:44.442563    2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:41:45.091351 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:45.091316    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:45.091541 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:45.091403    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx7bz" podUID="d3d2494a-0ef0-4da6-b6dd-225f750bb909"
Oct 30 02:41:45.091541 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:45.091505    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:45.091666 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:45.091624    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-skc8q" podUID="8a3ebba8-b807-4256-b552-5a18ec8412cb"
Oct 30 02:41:46.091458 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:46.091424    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:46.091916 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:46.091552    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5747" podUID="60924e48-a5e3-4f2b-ab35-48479c3b730b"
Oct 30 02:41:47.091689 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:47.091658    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:47.092036 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:47.091775    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx7bz" podUID="d3d2494a-0ef0-4da6-b6dd-225f750bb909"
Oct 30 02:41:47.092036 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:47.091826    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:47.092036 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:47.091905    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-skc8q" podUID="8a3ebba8-b807-4256-b552-5a18ec8412cb"
Oct 30 02:41:47.185710 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:47.185646    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:47.185836 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:47.185824    2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:41:47.196677 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:47.196652    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:41:48.091070 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.091048    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:48.091217 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:48.091147    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5747" podUID="60924e48-a5e3-4f2b-ab35-48479c3b730b"
Oct 30 02:41:48.373304 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.373242    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeReady"
Oct 30 02:41:48.373610 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.373366    2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 02:41:48.415866 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.415838    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2sq5m"]
Oct 30 02:41:48.420327 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.420303    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jsbwd"]
Oct 30 02:41:48.420821 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.420799    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2sq5m"
Oct 30 02:41:48.425096 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.425076    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"]
Oct 30 02:41:48.425249 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.425214    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jsbwd"
Oct 30 02:41:48.427990 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.427973    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dhf78"]
Oct 30 02:41:48.428151 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.428129    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.433044 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.433005    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-gxb72"]
Oct 30 02:41:48.435301 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.435280    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dhf78"
Oct 30 02:41:48.436222 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.436089    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"]
Oct 30 02:41:48.436344 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.436306    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.439018 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.438933    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5d4bdb4f7f-89p4x"]
Oct 30 02:41:48.439498 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.439052    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 02:41:48.439498 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.439126    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:41:48.442860 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.442464    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"]
Oct 30 02:41:48.442860 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.442646    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 02:41:48.443937 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.443485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:48.445545 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.445525    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"]
Oct 30 02:41:48.445662 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.445590    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"
Oct 30 02:41:48.448746 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.448726    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-njw6d"]
Oct 30 02:41:48.449626 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.449608    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:41:48.449724 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.449687    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 02:41:48.450674 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.449817    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 02:41:48.450674 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.449963    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 02:41:48.450674 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.450015    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8xpcs\""
Oct 30 02:41:48.453515 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.453498    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"]
Oct 30 02:41:48.453865 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.453687    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-njw6d"
Oct 30 02:41:48.461125 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.461106    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-scstg"]
Oct 30 02:41:48.461341 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.461325    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"
Oct 30 02:41:48.466676 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.466658    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c6dbd69f-lrwtw"]
Oct 30 02:41:48.466775 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.466760    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:41:48.473107 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.473084    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"]
Oct 30 02:41:48.473334 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.473317    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.477684 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.477663    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-7hpk4"]
Oct 30 02:41:48.477812 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.477792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:41:48.481525 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.481508    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7mhtz"]
Oct 30 02:41:48.481794 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.481619    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hpk4"
Oct 30 02:41:48.485466 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.485447    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fx6bk"]
Oct 30 02:41:48.485578 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.485561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:41:48.488384 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.488365    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2sq5m"]
Oct 30 02:41:48.488467 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.488430    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jsbwd"]
Oct 30 02:41:48.488535 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.488473    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"]
Oct 30 02:41:48.488535 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.488498    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"]
Oct 30 02:41:48.488535 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.488522    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-gxb72"]
Oct 30 02:41:48.488535 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.488526    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fx6bk"
Oct 30 02:41:48.488762 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.488544    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-njw6d"]
Oct 30 02:41:48.488762 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.488567    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5d4bdb4f7f-89p4x"]
Oct 30 02:41:48.488762 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.488589    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"]
Oct 30 02:41:48.488762 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.488616    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-scstg"]
Oct 30 02:41:48.488762 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.488636    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-7hpk4"]
Oct 30 02:41:48.488762 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.488658    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c6dbd69f-lrwtw"]
Oct 30 02:41:48.488762 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.488679    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dhf78"]
Oct 30 02:41:48.488762 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.488704    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"]
Oct 30 02:41:48.488762 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.488728    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"]
Oct 30 02:41:48.501697 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.501661    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7mhtz"]
Oct 30 02:41:48.505124 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.504901    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fx6bk"]
Oct 30 02:41:48.505124 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.504954    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"]
Oct 30 02:41:48.540677 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.540660    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 02:41:48.540866 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.540848    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 02:41:48.540866 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.540854    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 02:41:48.541023 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.540911    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 02:41:48.541139 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.541119    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 02:41:48.541303 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.541289    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 02:41:48.541648 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.541402    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 02:41:48.541648 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.541442    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:41:48.541648 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.541529    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 02:41:48.541648 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.541569    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:41:48.541648 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.541587    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 02:41:48.541648 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.541621    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 02:41:48.541648 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.541632    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:41:48.541648 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.541623    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:41:48.541648 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.541600    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 02:41:48.541648 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.541614    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:41:48.542126 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.541762    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 02:41:48.542395 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.542329    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 02:41:48.542395 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.542342    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 02:41:48.542395 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.542335    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 02:41:48.542395 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.542372    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:41:48.542587 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.542480    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 02:41:48.542587 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.542505    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 02:41:48.542587 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.542509    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:41:48.542587 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.542510    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 02:41:48.542587 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.542505    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 02:41:48.542744 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.542592    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 02:41:48.545161 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.545145    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 02:41:48.556458 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.556443    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 02:41:48.562009 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.561992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-ca-trust-extracted\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.562083 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-bound-sa-token\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.562083 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxg9s\" (UniqueName: \"kubernetes.io/projected/06715a40-fb77-4b9c-89e1-0aefe7b12a72-kube-api-access-jxg9s\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:41:48.562083 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562054    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.562172 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-service-ca-bundle\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.562172 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbq75\" (UniqueName: \"kubernetes.io/projected/cb0c5682-acc6-4029-831d-b6aa9027db25-kube-api-access-cbq75\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:48.562254 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-image-registry-private-configuration\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.562285 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d49cl\" (UniqueName: \"kubernetes.io/projected/d6516df6-4df4-4213-890a-0b25e274e606-kube-api-access-d49cl\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:41:48.562285 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-stats-auth\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:48.562369 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562304    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-image-registry-private-configuration\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.562369 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88d453b9-7e71-4561-8e1b-4b4074651675-serving-cert\") pod \"service-ca-operator-544d749c8c-vbz2s\" (UID: \"88d453b9-7e71-4561-8e1b-4b4074651675\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"
Oct 30 02:41:48.562467 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl6q6\" (UniqueName: \"kubernetes.io/projected/7e8de64a-fd25-4aa6-9160-b25e09d116aa-kube-api-access-wl6q6\") pod \"volume-data-source-validator-58c9f5f46d-njw6d\" (UID: \"7e8de64a-fd25-4aa6-9160-b25e09d116aa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-njw6d"
Oct 30 02:41:48.562518 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f697930c-fefc-42f2-835e-7b41cf115aaf-cert\") pod \"ingress-canary-fx6bk\" (UID: \"f697930c-fefc-42f2-835e-7b41cf115aaf\") " pod="openshift-ingress-canary/ingress-canary-fx6bk"
Oct 30 02:41:48.562568 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70f081e1-7202-4b86-930d-8c3f72a3bb7e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jwzx9\" (UID: \"70f081e1-7202-4b86-930d-8c3f72a3bb7e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:41:48.562568 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsrlt\" (UniqueName: \"kubernetes.io/projected/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-kube-api-access-tsrlt\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:41:48.562662 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562603    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/88d453b9-7e71-4561-8e1b-4b4074651675-config\") pod \"service-ca-operator-544d749c8c-vbz2s\" (UID: \"88d453b9-7e71-4561-8e1b-4b4074651675\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"
Oct 30 02:41:48.562662 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smpsr\" (UniqueName: \"kubernetes.io/projected/f697930c-fefc-42f2-835e-7b41cf115aaf-kube-api-access-smpsr\") pod \"ingress-canary-fx6bk\" (UID: \"f697930c-fefc-42f2-835e-7b41cf115aaf\") " pod="openshift-ingress-canary/ingress-canary-fx6bk"
Oct 30 02:41:48.562662 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.562783 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-serving-cert\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:41:48.562783 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562691    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-installation-pull-secrets\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.562783 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpm79\" (UniqueName: \"kubernetes.io/projected/0d2d9d6b-2705-49d3-b3d2-079d6d85148f-kube-api-access-tpm79\") pod \"kas-connection-checker-685895c7c9-jsbwd\" (UID: \"0d2d9d6b-2705-49d3-b3d2-079d6d85148f\") " pod="kube-system/kas-connection-checker-685895c7c9-jsbwd"
Oct 30 02:41:48.562783 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b93ba591-108f-41bd-afed-012dcd2f8690-ca-trust-extracted\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.562961 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-trusted-ca\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.562961 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c947k\" (UniqueName: \"kubernetes.io/projected/dfa424b6-593b-44ac-8e28-c9ba0e3207e8-kube-api-access-c947k\") pod \"network-check-source-6f78d59f99-7hpk4\" (UID: \"dfa424b6-593b-44ac-8e28-c9ba0e3207e8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hpk4"
Oct 30 02:41:48.562961 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562891    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg\" (UID: \"dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"
Oct 30 02:41:48.562961 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:41:48.563119 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562964    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-registry-certificates\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.563119 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.562991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmg6l\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-kube-api-access-hmg6l\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.563119 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:41:48.563119 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563055    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-tmp\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.563119 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:41:48.563317 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg\" (UID: \"dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"
Oct 30 02:41:48.563317 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563162    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svt6f\" (UniqueName: \"kubernetes.io/projected/88d453b9-7e71-4561-8e1b-4b4074651675-kube-api-access-svt6f\") pod \"service-ca-operator-544d749c8c-vbz2s\" (UID: \"88d453b9-7e71-4561-8e1b-4b4074651675\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"
Oct 30 02:41:48.563317 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-config-volume\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:41:48.563317 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:48.563317 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563282    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:41:48.563317 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.563317 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-trusted-ca\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.563569 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563330    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtmxt\" (UniqueName: \"kubernetes.io/projected/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-kube-api-access-vtmxt\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.563569 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb0c5682-acc6-4029-831d-b6aa9027db25-service-ca-bundle\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:48.563569 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-serving-cert\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.563569 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563467    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-installation-pull-secrets\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.563569 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs8x4\" (UniqueName: \"kubernetes.io/projected/c1cf080d-9f7d-4614-b446-76d174d9c3f7-kube-api-access-xs8x4\") pod \"kas-connection-checker-685895c7c9-dhf78\" (UID: \"c1cf080d-9f7d-4614-b446-76d174d9c3f7\") " pod="kube-system/kas-connection-checker-685895c7c9-dhf78"
Oct 30 02:41:48.563569 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-default-certificate\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:48.563569 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxvfq\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-kube-api-access-dxvfq\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.563831 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563566    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc5jg\" (UniqueName: \"kubernetes.io/projected/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8-kube-api-access-dc5jg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg\" (UID: \"dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"
Oct 30 02:41:48.563831 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-bound-sa-token\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.563831 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j6pt\" (UniqueName: \"kubernetes.io/projected/b1337da2-13ee-4b79-9a84-37ef217ae10f-kube-api-access-6j6pt\") pod \"kas-connection-checker-685895c7c9-2sq5m\" (UID: \"b1337da2-13ee-4b79-9a84-37ef217ae10f\") " pod="kube-system/kas-connection-checker-685895c7c9-2sq5m"
Oct 30 02:41:48.563831 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-config\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:41:48.563831 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-certificates\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.563831 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d6516df6-4df4-4213-890a-0b25e274e606-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:41:48.563831 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jwzx9\" (UID: \"70f081e1-7202-4b86-930d-8c3f72a3bb7e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:41:48.563831 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563741    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kfst\" (UniqueName: \"kubernetes.io/projected/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-kube-api-access-4kfst\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:41:48.563831 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-trusted-ca\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:41:48.564145 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-tmp-dir\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:41:48.564145 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.563866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-snapshots\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.569247 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.569224    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 02:41:48.569661 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.569636    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 02:41:48.569763 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.569722    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 02:41:48.569837 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.569822    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 02:41:48.569979 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.569967    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 02:41:48.570312 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.570297    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 02:41:48.570462 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.570441    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mcz9c\""
Oct 30 02:41:48.570515 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.570488    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 02:41:48.570647 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.570634    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 02:41:48.570700 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.570660    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 02:41:48.570850 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.570836    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 02:41:48.571027 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.571013    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-dgvq2\""
Oct 30 02:41:48.571257 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.571242    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qd44t\""
Oct 30 02:41:48.571375 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.571362    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 02:41:48.571500 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.571487    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 02:41:48.571579 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.571567    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 02:41:48.571753 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.571740    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hbwgl\""
Oct 30 02:41:48.571818 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.571782    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 02:41:48.572469 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.572447    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 02:41:48.664863 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.664811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:41:48.664863 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.664851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg\" (UID: \"dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"
Oct 30 02:41:48.665006 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.664870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svt6f\" (UniqueName: \"kubernetes.io/projected/88d453b9-7e71-4561-8e1b-4b4074651675-kube-api-access-svt6f\") pod \"service-ca-operator-544d749c8c-vbz2s\" (UID: \"88d453b9-7e71-4561-8e1b-4b4074651675\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"
Oct 30 02:41:48.665006 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.664886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-config-volume\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:41:48.665006 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.664910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:48.665006 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.664936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:41:48.665006 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:48.664946    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:41:48.665006 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.664961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.665006 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.664988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-trusted-ca\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.665006 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:48.665006    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls podName:d6516df6-4df4-4213-890a-0b25e274e606 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:49.164988088 +0000 UTC m=+30.578444212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vjvcf" (UID: "d6516df6-4df4-4213-890a-0b25e274e606") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:41:48.665425 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:48.665033    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:41:48.665425 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:48.665092    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs podName:cb0c5682-acc6-4029-831d-b6aa9027db25 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:49.165076321 +0000 UTC m=+30.578532432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs") pod "router-default-5d4bdb4f7f-89p4x" (UID: "cb0c5682-acc6-4029-831d-b6aa9027db25") : secret "router-metrics-certs-default" not found
Oct 30 02:41:48.665425 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.665038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtmxt\" (UniqueName: \"kubernetes.io/projected/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-kube-api-access-vtmxt\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.665425 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.665125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb0c5682-acc6-4029-831d-b6aa9027db25-service-ca-bundle\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:48.665425 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.665162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-serving-cert\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.665425 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.665229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-installation-pull-secrets\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.665425 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.665255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs8x4\" (UniqueName: \"kubernetes.io/projected/c1cf080d-9f7d-4614-b446-76d174d9c3f7-kube-api-access-xs8x4\") pod \"kas-connection-checker-685895c7c9-dhf78\" (UID: \"c1cf080d-9f7d-4614-b446-76d174d9c3f7\") " pod="kube-system/kas-connection-checker-685895c7c9-dhf78"
Oct 30 02:41:48.665425 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:48.665269    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb0c5682-acc6-4029-831d-b6aa9027db25-service-ca-bundle podName:cb0c5682-acc6-4029-831d-b6aa9027db25 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:49.165250639 +0000 UTC m=+30.578706775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cb0c5682-acc6-4029-831d-b6aa9027db25-service-ca-bundle") pod "router-default-5d4bdb4f7f-89p4x" (UID: "cb0c5682-acc6-4029-831d-b6aa9027db25") : configmap references non-existent config key: service-ca.crt
Oct 30 02:41:48.665425 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:48.665292    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:41:48.665425 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:48.665310    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:41:48.665425 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:48.665326    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c6dbd69f-lrwtw: secret "image-registry-tls" not found
Oct 30 02:41:48.665425 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:48.665333    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls podName:c5d42a36-81a2-49ad-9fa0-32d42e87d75d nodeName:}" failed. No retries permitted until 2026-10-30 02:41:49.165319913 +0000 UTC m=+30.578776041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls") pod "dns-default-7mhtz" (UID: "c5d42a36-81a2-49ad-9fa0-32d42e87d75d") : secret "dns-default-metrics-tls" not found
Oct 30 02:41:48.665425 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:48.665363    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls podName:b93ba591-108f-41bd-afed-012dcd2f8690 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:49.165349851 +0000 UTC m=+30.578805995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls") pod "image-registry-7c6dbd69f-lrwtw" (UID: "b93ba591-108f-41bd-afed-012dcd2f8690") : secret "image-registry-tls" not found
Oct 30 02:41:48.665425 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.665307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-default-certificate\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:48.665425 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.665401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxvfq\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-kube-api-access-dxvfq\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.666104 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.665433    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc5jg\" (UniqueName: \"kubernetes.io/projected/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8-kube-api-access-dc5jg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg\" (UID: \"dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"
Oct 30 02:41:48.666104 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.665458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-bound-sa-token\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.666104 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.665482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j6pt\" (UniqueName: \"kubernetes.io/projected/b1337da2-13ee-4b79-9a84-37ef217ae10f-kube-api-access-6j6pt\") pod \"kas-connection-checker-685895c7c9-2sq5m\" (UID: \"b1337da2-13ee-4b79-9a84-37ef217ae10f\") " pod="kube-system/kas-connection-checker-685895c7c9-2sq5m"
Oct 30 02:41:48.666104 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.665508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-config\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:41:48.666104 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.665549    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg\" (UID: \"dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"
Oct 30 02:41:48.666104 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.665558    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-config-volume\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:41:48.666104 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.665895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-certificates\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.666104 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.665932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d6516df6-4df4-4213-890a-0b25e274e606-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:41:48.666104 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.665960    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jwzx9\" (UID: \"70f081e1-7202-4b86-930d-8c3f72a3bb7e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:41:48.666104 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.665987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kfst\" (UniqueName: \"kubernetes.io/projected/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-kube-api-access-4kfst\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:41:48.666104 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-trusted-ca\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:41:48.666104 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-tmp-dir\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:41:48.666104 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666064    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-trusted-ca\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.666104 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-snapshots\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.666801 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-ca-trust-extracted\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.666801 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-bound-sa-token\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.666801 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxg9s\" (UniqueName: \"kubernetes.io/projected/06715a40-fb77-4b9c-89e1-0aefe7b12a72-kube-api-access-jxg9s\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:41:48.666801 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.666801 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666261    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-service-ca-bundle\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.666801 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbq75\" (UniqueName: \"kubernetes.io/projected/cb0c5682-acc6-4029-831d-b6aa9027db25-kube-api-access-cbq75\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:48.666801 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-image-registry-private-configuration\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.666801 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d49cl\" (UniqueName: \"kubernetes.io/projected/d6516df6-4df4-4213-890a-0b25e274e606-kube-api-access-d49cl\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:41:48.666801 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666360    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-certificates\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.666801 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-stats-auth\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:48.666801 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666397    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-config\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:41:48.666801 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-image-registry-private-configuration\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.666801 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88d453b9-7e71-4561-8e1b-4b4074651675-serving-cert\") pod \"service-ca-operator-544d749c8c-vbz2s\" (UID: \"88d453b9-7e71-4561-8e1b-4b4074651675\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"
Oct 30 02:41:48.666801 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl6q6\" (UniqueName: \"kubernetes.io/projected/7e8de64a-fd25-4aa6-9160-b25e09d116aa-kube-api-access-wl6q6\") pod \"volume-data-source-validator-58c9f5f46d-njw6d\" (UID: \"7e8de64a-fd25-4aa6-9160-b25e09d116aa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-njw6d"
Oct 30 02:41:48.666801 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666505    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f697930c-fefc-42f2-835e-7b41cf115aaf-cert\") pod \"ingress-canary-fx6bk\" (UID: \"f697930c-fefc-42f2-835e-7b41cf115aaf\") " pod="openshift-ingress-canary/ingress-canary-fx6bk"
Oct 30 02:41:48.666801 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666529    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70f081e1-7202-4b86-930d-8c3f72a3bb7e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jwzx9\" (UID: \"70f081e1-7202-4b86-930d-8c3f72a3bb7e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:41:48.669883 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666554    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsrlt\" (UniqueName: \"kubernetes.io/projected/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-kube-api-access-tsrlt\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:41:48.669883 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666561    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-snapshots\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.669883 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/88d453b9-7e71-4561-8e1b-4b4074651675-config\") pod \"service-ca-operator-544d749c8c-vbz2s\" (UID: \"88d453b9-7e71-4561-8e1b-4b4074651675\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"
Oct 30 02:41:48.669883 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smpsr\" (UniqueName: \"kubernetes.io/projected/f697930c-fefc-42f2-835e-7b41cf115aaf-kube-api-access-smpsr\") pod \"ingress-canary-fx6bk\" (UID: \"f697930c-fefc-42f2-835e-7b41cf115aaf\") " pod="openshift-ingress-canary/ingress-canary-fx6bk"
Oct 30 02:41:48.669883 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.669883 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-serving-cert\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:41:48.669883 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-installation-pull-secrets\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.669883 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpm79\" (UniqueName: \"kubernetes.io/projected/0d2d9d6b-2705-49d3-b3d2-079d6d85148f-kube-api-access-tpm79\") pod \"kas-connection-checker-685895c7c9-jsbwd\" (UID: \"0d2d9d6b-2705-49d3-b3d2-079d6d85148f\") " pod="kube-system/kas-connection-checker-685895c7c9-jsbwd"
Oct 30 02:41:48.669883 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b93ba591-108f-41bd-afed-012dcd2f8690-ca-trust-extracted\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.669883 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-trusted-ca\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.669883 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666810    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-tmp-dir\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:41:48.669883 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666818    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c947k\" (UniqueName: \"kubernetes.io/projected/dfa424b6-593b-44ac-8e28-c9ba0e3207e8-kube-api-access-c947k\") pod \"network-check-source-6f78d59f99-7hpk4\" (UID: \"dfa424b6-593b-44ac-8e28-c9ba0e3207e8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hpk4"
Oct 30 02:41:48.669883 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg\" (UID: \"dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"
Oct 30 02:41:48.669883 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:48.666892    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:41:48.669883 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:48.666906    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59bd5b5c6b-lcbtn: secret "image-registry-tls" not found
Oct 30 02:41:48.669883 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:48.666945    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls podName:bedee86a-79f7-43cf-9ed2-5dbf5b8109eb nodeName:}" failed. No retries permitted until 2026-10-30 02:41:49.166931687 +0000 UTC m=+30.580387814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls") pod "image-registry-59bd5b5c6b-lcbtn" (UID: "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb") : secret "image-registry-tls" not found
Oct 30 02:41:48.669883 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:48.666967    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:41:48.670891 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:48.667010    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls podName:06715a40-fb77-4b9c-89e1-0aefe7b12a72 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:49.166997152 +0000 UTC m=+30.580453279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7mfqp" (UID: "06715a40-fb77-4b9c-89e1-0aefe7b12a72") : secret "samples-operator-tls" not found
Oct 30 02:41:48.670891 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.667281    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-ca-trust-extracted\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.670891 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.667395    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d6516df6-4df4-4213-890a-0b25e274e606-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:41:48.670891 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.666901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:41:48.670891 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.667452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-registry-certificates\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.670891 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.667482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmg6l\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-kube-api-access-hmg6l\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.670891 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:48.667491    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:41:48.670891 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.667515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:41:48.670891 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:48.667540    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert podName:70f081e1-7202-4b86-930d-8c3f72a3bb7e nodeName:}" failed. No retries permitted until 2026-10-30 02:41:49.167525453 +0000 UTC m=+30.580981580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jwzx9" (UID: "70f081e1-7202-4b86-930d-8c3f72a3bb7e") : secret "networking-console-plugin-cert" not found
Oct 30 02:41:48.670891 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.667574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-tmp\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.670891 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.667598    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-service-ca-bundle\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.670891 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.668079    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-tmp\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.670891 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.668090    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-trusted-ca\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:41:48.670891 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.668368    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:41:48.670891 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.668653    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-registry-certificates\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.671609 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.668994    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b93ba591-108f-41bd-afed-012dcd2f8690-ca-trust-extracted\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.671609 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.670257    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-serving-cert\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.671609 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.670527    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.671609 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.670659    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-trusted-ca\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.671609 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.670931    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-image-registry-private-configuration\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.671609 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.671473    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70f081e1-7202-4b86-930d-8c3f72a3bb7e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jwzx9\" (UID: \"70f081e1-7202-4b86-930d-8c3f72a3bb7e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:41:48.671890 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.671726    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/88d453b9-7e71-4561-8e1b-4b4074651675-config\") pod \"service-ca-operator-544d749c8c-vbz2s\" (UID: \"88d453b9-7e71-4561-8e1b-4b4074651675\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"
Oct 30 02:41:48.672151 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.672105    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-installation-pull-secrets\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.672361 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.672309    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-default-certificate\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:48.672556 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.672533    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88d453b9-7e71-4561-8e1b-4b4074651675-serving-cert\") pod \"service-ca-operator-544d749c8c-vbz2s\" (UID: \"88d453b9-7e71-4561-8e1b-4b4074651675\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"
Oct 30 02:41:48.672784 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.672698    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:48.672947 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.672926    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-stats-auth\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:48.673010 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.672987    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-installation-pull-secrets\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.673354 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.673326    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-serving-cert\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:41:48.673818 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.673797    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg\" (UID: \"dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"
Oct 30 02:41:48.674070 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.674052    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f697930c-fefc-42f2-835e-7b41cf115aaf-cert\") pod \"ingress-canary-fx6bk\" (UID: \"f697930c-fefc-42f2-835e-7b41cf115aaf\") " pod="openshift-ingress-canary/ingress-canary-fx6bk"
Oct 30 02:41:48.674447 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.674430    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-image-registry-private-configuration\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.686476 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.686454    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svt6f\" (UniqueName: \"kubernetes.io/projected/88d453b9-7e71-4561-8e1b-4b4074651675-kube-api-access-svt6f\") pod \"service-ca-operator-544d749c8c-vbz2s\" (UID: \"88d453b9-7e71-4561-8e1b-4b4074651675\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"
Oct 30 02:41:48.688137 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.688005    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs8x4\" (UniqueName: \"kubernetes.io/projected/c1cf080d-9f7d-4614-b446-76d174d9c3f7-kube-api-access-xs8x4\") pod \"kas-connection-checker-685895c7c9-dhf78\" (UID: \"c1cf080d-9f7d-4614-b446-76d174d9c3f7\") " pod="kube-system/kas-connection-checker-685895c7c9-dhf78"
Oct 30 02:41:48.688783 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.688726    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtmxt\" (UniqueName: \"kubernetes.io/projected/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-kube-api-access-vtmxt\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.688873 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.688787    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j6pt\" (UniqueName: \"kubernetes.io/projected/b1337da2-13ee-4b79-9a84-37ef217ae10f-kube-api-access-6j6pt\") pod \"kas-connection-checker-685895c7c9-2sq5m\" (UID: \"b1337da2-13ee-4b79-9a84-37ef217ae10f\") " pod="kube-system/kas-connection-checker-685895c7c9-2sq5m"
Oct 30 02:41:48.689404 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.689361    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kfst\" (UniqueName: \"kubernetes.io/projected/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-kube-api-access-4kfst\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:41:48.690519 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.690499    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc5jg\" (UniqueName: \"kubernetes.io/projected/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8-kube-api-access-dc5jg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg\" (UID: \"dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"
Oct 30 02:41:48.690839 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.690809    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-bound-sa-token\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.691279 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.691262    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxg9s\" (UniqueName: \"kubernetes.io/projected/06715a40-fb77-4b9c-89e1-0aefe7b12a72-kube-api-access-jxg9s\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:41:48.691426 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.691403    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxvfq\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-kube-api-access-dxvfq\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:48.692280 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.692256    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-bound-sa-token\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.693120 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.693024    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smpsr\" (UniqueName: \"kubernetes.io/projected/f697930c-fefc-42f2-835e-7b41cf115aaf-kube-api-access-smpsr\") pod \"ingress-canary-fx6bk\" (UID: \"f697930c-fefc-42f2-835e-7b41cf115aaf\") " pod="openshift-ingress-canary/ingress-canary-fx6bk"
Oct 30 02:41:48.693120 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.693075    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmg6l\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-kube-api-access-hmg6l\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:48.693745 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.693702    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d49cl\" (UniqueName: \"kubernetes.io/projected/d6516df6-4df4-4213-890a-0b25e274e606-kube-api-access-d49cl\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:41:48.694754 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.694733    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsrlt\" (UniqueName: \"kubernetes.io/projected/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-kube-api-access-tsrlt\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:41:48.695261 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.695240    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpm79\" (UniqueName: \"kubernetes.io/projected/0d2d9d6b-2705-49d3-b3d2-079d6d85148f-kube-api-access-tpm79\") pod \"kas-connection-checker-685895c7c9-jsbwd\" (UID: \"0d2d9d6b-2705-49d3-b3d2-079d6d85148f\") " pod="kube-system/kas-connection-checker-685895c7c9-jsbwd"
Oct 30 02:41:48.695466 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.695449    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbq75\" (UniqueName: \"kubernetes.io/projected/cb0c5682-acc6-4029-831d-b6aa9027db25-kube-api-access-cbq75\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:48.695624 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.695606    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl6q6\" (UniqueName: \"kubernetes.io/projected/7e8de64a-fd25-4aa6-9160-b25e09d116aa-kube-api-access-wl6q6\") pod \"volume-data-source-validator-58c9f5f46d-njw6d\" (UID: \"7e8de64a-fd25-4aa6-9160-b25e09d116aa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-njw6d"
Oct 30 02:41:48.695848 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.695830    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c947k\" (UniqueName: \"kubernetes.io/projected/dfa424b6-593b-44ac-8e28-c9ba0e3207e8-kube-api-access-c947k\") pod \"network-check-source-6f78d59f99-7hpk4\" (UID: \"dfa424b6-593b-44ac-8e28-c9ba0e3207e8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hpk4"
Oct 30 02:41:48.741297 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.741269    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2sq5m"
Oct 30 02:41:48.761808 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.761673    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jsbwd"
Oct 30 02:41:48.779584 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.779557    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dhf78"
Oct 30 02:41:48.789035 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.788883    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:41:48.810897 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.810457    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"
Oct 30 02:41:48.823972 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.823588    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-njw6d"
Oct 30 02:41:48.833319 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.833123    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"
Oct 30 02:41:48.835875 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.835371    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:48.835875 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.835509    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:41:48.853025 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.852953    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hpk4"
Oct 30 02:41:48.867268 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:48.867000    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fx6bk"
Oct 30 02:41:49.049641 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.049620    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2sq5m"]
Oct 30 02:41:49.051145 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:49.051122    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1337da2_13ee_4b79_9a84_37ef217ae10f.slice/crio-66b0866bd0781a66cf486595a939b5ec710e86dc87d6021b5f57cbff2c2f2a7a WatchSource:0}: Error finding container 66b0866bd0781a66cf486595a939b5ec710e86dc87d6021b5f57cbff2c2f2a7a: Status 404 returned error can't find the container with id 66b0866bd0781a66cf486595a939b5ec710e86dc87d6021b5f57cbff2c2f2a7a
Oct 30 02:41:49.063943 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.063923    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jsbwd"]
Oct 30 02:41:49.065431 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:49.065378    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2d9d6b_2705_49d3_b3d2_079d6d85148f.slice/crio-e6fc49c2f7adc853e0db6bde219654ea6290fed8a8b397ef2ac48964c122a386 WatchSource:0}: Error finding container e6fc49c2f7adc853e0db6bde219654ea6290fed8a8b397ef2ac48964c122a386: Status 404 returned error can't find the container with id e6fc49c2f7adc853e0db6bde219654ea6290fed8a8b397ef2ac48964c122a386
Oct 30 02:41:49.091119 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.090969    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:49.093811 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.093481    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:49.109788 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.109097    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-gxb72"]
Oct 30 02:41:49.111244 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.111161    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dhf78"]
Oct 30 02:41:49.112246 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:49.112220    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e46a1b_7b03_4ce0_aa6e_ea5881c0a1d9.slice/crio-91181e4e17e08fed6169aeca64410d61a4ff599dcd8c981d467db052e2d24cdb WatchSource:0}: Error finding container 91181e4e17e08fed6169aeca64410d61a4ff599dcd8c981d467db052e2d24cdb: Status 404 returned error can't find the container with id 91181e4e17e08fed6169aeca64410d61a4ff599dcd8c981d467db052e2d24cdb
Oct 30 02:41:49.112572 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.112552    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-d89pf\""
Oct 30 02:41:49.112688 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.112601    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j74dx\""
Oct 30 02:41:49.112688 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.112636    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:41:49.116689 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:49.116663    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1cf080d_9f7d_4614_b446_76d174d9c3f7.slice/crio-180be8420badf19cba5a9a9e962f858c4bc29b011327502f30c53c0a27010815 WatchSource:0}: Error finding container 180be8420badf19cba5a9a9e962f858c4bc29b011327502f30c53c0a27010815: Status 404 returned error can't find the container with id 180be8420badf19cba5a9a9e962f858c4bc29b011327502f30c53c0a27010815
Oct 30 02:41:49.168207 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.165496    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"]
Oct 30 02:41:49.172063 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.172016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:41:49.172138 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.172074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:41:49.172138 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.172118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:49.172227 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.172150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:41:49.172227 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.172174    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:49.172227 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.172221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb0c5682-acc6-4029-831d-b6aa9027db25-service-ca-bundle\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:49.174242 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.172280    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jwzx9\" (UID: \"70f081e1-7202-4b86-930d-8c3f72a3bb7e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:41:49.174242 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.172320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:49.174242 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:49.172472    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:41:49.174242 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:49.172487    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59bd5b5c6b-lcbtn: secret "image-registry-tls" not found
Oct 30 02:41:49.174242 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:49.172540    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls podName:bedee86a-79f7-43cf-9ed2-5dbf5b8109eb nodeName:}" failed. No retries permitted until 2026-10-30 02:41:50.172521574 +0000 UTC m=+31.585977703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls") pod "image-registry-59bd5b5c6b-lcbtn" (UID: "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb") : secret "image-registry-tls" not found
Oct 30 02:41:49.174242 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:49.172879    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:41:49.174242 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:49.172925    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls podName:06715a40-fb77-4b9c-89e1-0aefe7b12a72 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:50.172911485 +0000 UTC m=+31.586367610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7mfqp" (UID: "06715a40-fb77-4b9c-89e1-0aefe7b12a72") : secret "samples-operator-tls" not found
Oct 30 02:41:49.174242 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:49.172994    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:41:49.174242 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:49.173030    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls podName:d6516df6-4df4-4213-890a-0b25e274e606 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:50.173019037 +0000 UTC m=+31.586475164 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vjvcf" (UID: "d6516df6-4df4-4213-890a-0b25e274e606") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:41:49.174242 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:49.173082    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:41:49.174242 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:49.173111    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs podName:cb0c5682-acc6-4029-831d-b6aa9027db25 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:50.173101115 +0000 UTC m=+31.586557238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs") pod "router-default-5d4bdb4f7f-89p4x" (UID: "cb0c5682-acc6-4029-831d-b6aa9027db25") : secret "router-metrics-certs-default" not found
Oct 30 02:41:49.174242 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:49.173171    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:41:49.174242 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:49.173215    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls podName:c5d42a36-81a2-49ad-9fa0-32d42e87d75d nodeName:}" failed. No retries permitted until 2026-10-30 02:41:50.173205236 +0000 UTC m=+31.586661359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls") pod "dns-default-7mhtz" (UID: "c5d42a36-81a2-49ad-9fa0-32d42e87d75d") : secret "dns-default-metrics-tls" not found
Oct 30 02:41:49.174242 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:49.173271    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:41:49.174242 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:49.173281    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c6dbd69f-lrwtw: secret "image-registry-tls" not found
Oct 30 02:41:49.174242 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:49.173310    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls podName:b93ba591-108f-41bd-afed-012dcd2f8690 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:50.1733001 +0000 UTC m=+31.586756224 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls") pod "image-registry-7c6dbd69f-lrwtw" (UID: "b93ba591-108f-41bd-afed-012dcd2f8690") : secret "image-registry-tls" not found
Oct 30 02:41:49.175419 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:49.173369    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb0c5682-acc6-4029-831d-b6aa9027db25-service-ca-bundle podName:cb0c5682-acc6-4029-831d-b6aa9027db25 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:50.173360214 +0000 UTC m=+31.586816338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cb0c5682-acc6-4029-831d-b6aa9027db25-service-ca-bundle") pod "router-default-5d4bdb4f7f-89p4x" (UID: "cb0c5682-acc6-4029-831d-b6aa9027db25") : configmap references non-existent config key: service-ca.crt
Oct 30 02:41:49.175419 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:49.173420    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:41:49.175419 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:49.173451    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert podName:70f081e1-7202-4b86-930d-8c3f72a3bb7e nodeName:}" failed. No retries permitted until 2026-10-30 02:41:50.17343982 +0000 UTC m=+31.586895943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jwzx9" (UID: "70f081e1-7202-4b86-930d-8c3f72a3bb7e") : secret "networking-console-plugin-cert" not found
Oct 30 02:41:49.179369 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.179278    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-scstg"]
Oct 30 02:41:49.179369 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.179328    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"]
Oct 30 02:41:49.179956 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:49.179928    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d453b9_7e71_4561_8e1b_4b4074651675.slice/crio-753ec90d7ab7e4722c76de269ba3c7110ee43e363062e9fd3e563965414d2bda WatchSource:0}: Error finding container 753ec90d7ab7e4722c76de269ba3c7110ee43e363062e9fd3e563965414d2bda: Status 404 returned error can't find the container with id 753ec90d7ab7e4722c76de269ba3c7110ee43e363062e9fd3e563965414d2bda
Oct 30 02:41:49.184853 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:49.184460    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd14a05e_69c0_4a9a_99d8_0ff750ceb3c8.slice/crio-02d8d231e9a19a0f506667e517f3cdf3993ea52b4b74c83b8a88a120b220d04c WatchSource:0}: Error finding container 02d8d231e9a19a0f506667e517f3cdf3993ea52b4b74c83b8a88a120b220d04c: Status 404 returned error can't find the container with id 02d8d231e9a19a0f506667e517f3cdf3993ea52b4b74c83b8a88a120b220d04c
Oct 30 02:41:49.186881 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:49.186843    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb37ddc9_9a43_46c5_90e5_e8244f9dfe9e.slice/crio-e7a3b1ae985324333bafc677eb2ce527fe499935b2c7ab6fabc34cc6f7b16e37 WatchSource:0}: Error finding container e7a3b1ae985324333bafc677eb2ce527fe499935b2c7ab6fabc34cc6f7b16e37: Status 404 returned error can't find the container with id e7a3b1ae985324333bafc677eb2ce527fe499935b2c7ab6fabc34cc6f7b16e37
Oct 30 02:41:49.204481 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.201349    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-njw6d"]
Oct 30 02:41:49.204584 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:49.204538    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8de64a_fd25_4aa6_9160_b25e09d116aa.slice/crio-e31c17097aecc157404375a3654588a06aae2b3b39f29448f0da4bae02470c79 WatchSource:0}: Error finding container e31c17097aecc157404375a3654588a06aae2b3b39f29448f0da4bae02470c79: Status 404 returned error can't find the container with id e31c17097aecc157404375a3654588a06aae2b3b39f29448f0da4bae02470c79
Oct 30 02:41:49.215074 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.215043    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-7hpk4"]
Oct 30 02:41:49.218386 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:49.218352    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa424b6_593b_44ac_8e28_c9ba0e3207e8.slice/crio-5668c9636dbb9aff7816e1059ac0e651e49fb53431a60899b0f30e2240b0c1bb WatchSource:0}: Error finding container 5668c9636dbb9aff7816e1059ac0e651e49fb53431a60899b0f30e2240b0c1bb: Status 404 returned error can't find the container with id 5668c9636dbb9aff7816e1059ac0e651e49fb53431a60899b0f30e2240b0c1bb
Oct 30 02:41:49.222169 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.222103    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fx6bk"]
Oct 30 02:41:49.225776 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:49.225640    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf697930c_fefc_42f2_835e_7b41cf115aaf.slice/crio-410a6e7ef4e47afad48737d7e85659ac5aa54e1fb2e993f98cfd49d5410a53d8 WatchSource:0}: Error finding container 410a6e7ef4e47afad48737d7e85659ac5aa54e1fb2e993f98cfd49d5410a53d8: Status 404 returned error can't find the container with id 410a6e7ef4e47afad48737d7e85659ac5aa54e1fb2e993f98cfd49d5410a53d8
Oct 30 02:41:49.456363 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.456278    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg" event={"ID":"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e","Type":"ContainerStarted","Data":"e7a3b1ae985324333bafc677eb2ce527fe499935b2c7ab6fabc34cc6f7b16e37"}
Oct 30 02:41:49.457408 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.457386    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s" event={"ID":"88d453b9-7e71-4561-8e1b-4b4074651675","Type":"ContainerStarted","Data":"753ec90d7ab7e4722c76de269ba3c7110ee43e363062e9fd3e563965414d2bda"}
Oct 30 02:41:49.458482 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.458463    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-gxb72" event={"ID":"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9","Type":"ContainerStarted","Data":"91181e4e17e08fed6169aeca64410d61a4ff599dcd8c981d467db052e2d24cdb"}
Oct 30 02:41:49.459653 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.459635    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fx6bk" event={"ID":"f697930c-fefc-42f2-835e-7b41cf115aaf","Type":"ContainerStarted","Data":"410a6e7ef4e47afad48737d7e85659ac5aa54e1fb2e993f98cfd49d5410a53d8"}
Oct 30 02:41:49.460774 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.460755    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hpk4" event={"ID":"dfa424b6-593b-44ac-8e28-c9ba0e3207e8","Type":"ContainerStarted","Data":"5668c9636dbb9aff7816e1059ac0e651e49fb53431a60899b0f30e2240b0c1bb"}
Oct 30 02:41:49.461851 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.461829    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg" event={"ID":"dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8","Type":"ContainerStarted","Data":"02d8d231e9a19a0f506667e517f3cdf3993ea52b4b74c83b8a88a120b220d04c"}
Oct 30 02:41:49.465562 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.465540    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dhf78" event={"ID":"c1cf080d-9f7d-4614-b446-76d174d9c3f7","Type":"ContainerStarted","Data":"03735a3224709797ad085140331227c87668ba993f925dd7a2a109cfc88ff040"}
Oct 30 02:41:49.465662 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.465580    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dhf78" event={"ID":"c1cf080d-9f7d-4614-b446-76d174d9c3f7","Type":"ContainerStarted","Data":"180be8420badf19cba5a9a9e962f858c4bc29b011327502f30c53c0a27010815"}
Oct 30 02:41:49.467280 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.467256    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2sq5m" event={"ID":"b1337da2-13ee-4b79-9a84-37ef217ae10f","Type":"ContainerStarted","Data":"b5d61cc32394ccfb5f351952ab453bdda79f264e5b001b3193f89fbfd2c958ce"}
Oct 30 02:41:49.467383 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.467290    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2sq5m" event={"ID":"b1337da2-13ee-4b79-9a84-37ef217ae10f","Type":"ContainerStarted","Data":"66b0866bd0781a66cf486595a939b5ec710e86dc87d6021b5f57cbff2c2f2a7a"}
Oct 30 02:41:49.468437 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.468415    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-njw6d" event={"ID":"7e8de64a-fd25-4aa6-9160-b25e09d116aa","Type":"ContainerStarted","Data":"e31c17097aecc157404375a3654588a06aae2b3b39f29448f0da4bae02470c79"}
Oct 30 02:41:49.469919 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.469898    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jsbwd" event={"ID":"0d2d9d6b-2705-49d3-b3d2-079d6d85148f","Type":"ContainerStarted","Data":"5180e4e1b4dbc746a1b69fa7b839b4863bea1175865a67fd17e5783092dc2908"}
Oct 30 02:41:49.469991 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.469934    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jsbwd" event={"ID":"0d2d9d6b-2705-49d3-b3d2-079d6d85148f","Type":"ContainerStarted","Data":"e6fc49c2f7adc853e0db6bde219654ea6290fed8a8b397ef2ac48964c122a386"}
Oct 30 02:41:49.507602 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.507554    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-dhf78" podStartSLOduration=483.507539523 podStartE2EDuration="8m3.507539523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:49.507199195 +0000 UTC m=+30.920655328" watchObservedRunningTime="2026-10-30 02:41:49.507539523 +0000 UTC m=+30.920995657"
Oct 30 02:41:49.549680 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.549642    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-2sq5m" podStartSLOduration=483.549631267 podStartE2EDuration="8m3.549631267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:49.548525712 +0000 UTC m=+30.961981841" watchObservedRunningTime="2026-10-30 02:41:49.549631267 +0000 UTC m=+30.963087398"
Oct 30 02:41:49.590011 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:49.589976    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-jsbwd" podStartSLOduration=483.589964879 podStartE2EDuration="8m3.589964879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:49.589317987 +0000 UTC m=+31.002774116" watchObservedRunningTime="2026-10-30 02:41:49.589964879 +0000 UTC m=+31.003421014"
Oct 30 02:41:50.091135 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:50.091100    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:50.107642 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:50.107616    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:41:50.107904 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:50.107886    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-czntg\""
Oct 30 02:41:50.182081 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:50.182041    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:41:50.182251 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:50.182118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:50.182251 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:50.182153    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:41:50.182446 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:50.182395    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:41:50.182531 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:50.182467    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs podName:cb0c5682-acc6-4029-831d-b6aa9027db25 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:52.182446881 +0000 UTC m=+33.595903015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs") pod "router-default-5d4bdb4f7f-89p4x" (UID: "cb0c5682-acc6-4029-831d-b6aa9027db25") : secret "router-metrics-certs-default" not found
Oct 30 02:41:50.182591 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:50.182547    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:41:50.182639 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:50.182589    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls podName:d6516df6-4df4-4213-890a-0b25e274e606 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:52.182576774 +0000 UTC m=+33.596032907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vjvcf" (UID: "d6516df6-4df4-4213-890a-0b25e274e606") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:41:50.183010 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:50.182985    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:41:50.183081 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:50.183044    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls podName:c5d42a36-81a2-49ad-9fa0-32d42e87d75d nodeName:}" failed. No retries permitted until 2026-10-30 02:41:52.18302823 +0000 UTC m=+33.596484355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls") pod "dns-default-7mhtz" (UID: "c5d42a36-81a2-49ad-9fa0-32d42e87d75d") : secret "dns-default-metrics-tls" not found
Oct 30 02:41:50.183144 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:50.183102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:50.183229 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:50.183165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb0c5682-acc6-4029-831d-b6aa9027db25-service-ca-bundle\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:50.183288 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:50.183253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jwzx9\" (UID: \"70f081e1-7202-4b86-930d-8c3f72a3bb7e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:41:50.183346 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:50.183298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:50.183401 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:50.183371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:41:50.183502 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:50.183485    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:41:50.183559 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:50.183537    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls podName:06715a40-fb77-4b9c-89e1-0aefe7b12a72 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:52.183523241 +0000 UTC m=+33.596979365 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7mfqp" (UID: "06715a40-fb77-4b9c-89e1-0aefe7b12a72") : secret "samples-operator-tls" not found
Oct 30 02:41:50.183615 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:50.183607    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:41:50.183667 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:50.183619    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c6dbd69f-lrwtw: secret "image-registry-tls" not found
Oct 30 02:41:50.183667 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:50.183655    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls podName:b93ba591-108f-41bd-afed-012dcd2f8690 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:52.183643899 +0000 UTC m=+33.597100028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls") pod "image-registry-7c6dbd69f-lrwtw" (UID: "b93ba591-108f-41bd-afed-012dcd2f8690") : secret "image-registry-tls" not found
Oct 30 02:41:50.183773 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:50.183728    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb0c5682-acc6-4029-831d-b6aa9027db25-service-ca-bundle podName:cb0c5682-acc6-4029-831d-b6aa9027db25 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:52.183717243 +0000 UTC m=+33.597173366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cb0c5682-acc6-4029-831d-b6aa9027db25-service-ca-bundle") pod "router-default-5d4bdb4f7f-89p4x" (UID: "cb0c5682-acc6-4029-831d-b6aa9027db25") : configmap references non-existent config key: service-ca.crt
Oct 30 02:41:50.183828 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:50.183787    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:41:50.183828 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:50.183824    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert podName:70f081e1-7202-4b86-930d-8c3f72a3bb7e nodeName:}" failed. No retries permitted until 2026-10-30 02:41:52.183812789 +0000 UTC m=+33.597268914 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jwzx9" (UID: "70f081e1-7202-4b86-930d-8c3f72a3bb7e") : secret "networking-console-plugin-cert" not found
Oct 30 02:41:50.183925 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:50.183887    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:41:50.183925 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:50.183898    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59bd5b5c6b-lcbtn: secret "image-registry-tls" not found
Oct 30 02:41:50.183998 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:50.183931    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls podName:bedee86a-79f7-43cf-9ed2-5dbf5b8109eb nodeName:}" failed. No retries permitted until 2026-10-30 02:41:52.183920374 +0000 UTC m=+33.597376499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls") pod "image-registry-59bd5b5c6b-lcbtn" (UID: "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb") : secret "image-registry-tls" not found
Oct 30 02:41:50.410756 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:50.410676    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:52.205589 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:52.204023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:52.205589 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:52.204102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:41:52.205589 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:52.204146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:41:52.205589 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:52.204203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:52.205589 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:52.204234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:41:52.205589 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:52.204257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:52.205589 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:52.204278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb0c5682-acc6-4029-831d-b6aa9027db25-service-ca-bundle\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:52.205589 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:52.204330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jwzx9\" (UID: \"70f081e1-7202-4b86-930d-8c3f72a3bb7e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:41:52.205589 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:52.204446    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:41:52.205589 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:52.204504    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert podName:70f081e1-7202-4b86-930d-8c3f72a3bb7e nodeName:}" failed. No retries permitted until 2026-10-30 02:41:56.204492046 +0000 UTC m=+37.617948169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jwzx9" (UID: "70f081e1-7202-4b86-930d-8c3f72a3bb7e") : secret "networking-console-plugin-cert" not found
Oct 30 02:41:52.205589 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:52.204893    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:41:52.205589 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:52.204905    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59bd5b5c6b-lcbtn: secret "image-registry-tls" not found
Oct 30 02:41:52.205589 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:52.204948    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls podName:bedee86a-79f7-43cf-9ed2-5dbf5b8109eb nodeName:}" failed. No retries permitted until 2026-10-30 02:41:56.204933628 +0000 UTC m=+37.618389754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls") pod "image-registry-59bd5b5c6b-lcbtn" (UID: "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb") : secret "image-registry-tls" not found
Oct 30 02:41:52.205589 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:52.205033    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:41:52.205589 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:52.205077    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls podName:06715a40-fb77-4b9c-89e1-0aefe7b12a72 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:56.205064902 +0000 UTC m=+37.618521025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7mfqp" (UID: "06715a40-fb77-4b9c-89e1-0aefe7b12a72") : secret "samples-operator-tls" not found
Oct 30 02:41:52.205589 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:52.205140    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:41:52.206992 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:52.205199    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls podName:d6516df6-4df4-4213-890a-0b25e274e606 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:56.205168059 +0000 UTC m=+37.618624216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vjvcf" (UID: "d6516df6-4df4-4213-890a-0b25e274e606") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:41:52.206992 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:52.205259    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:41:52.206992 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:52.205298    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs podName:cb0c5682-acc6-4029-831d-b6aa9027db25 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:56.205286714 +0000 UTC m=+37.618742838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs") pod "router-default-5d4bdb4f7f-89p4x" (UID: "cb0c5682-acc6-4029-831d-b6aa9027db25") : secret "router-metrics-certs-default" not found
Oct 30 02:41:52.206992 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:52.205355    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:41:52.206992 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:52.205378    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls podName:c5d42a36-81a2-49ad-9fa0-32d42e87d75d nodeName:}" failed. No retries permitted until 2026-10-30 02:41:56.205372139 +0000 UTC m=+37.618828262 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls") pod "dns-default-7mhtz" (UID: "c5d42a36-81a2-49ad-9fa0-32d42e87d75d") : secret "dns-default-metrics-tls" not found
Oct 30 02:41:52.206992 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:52.205430    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:41:52.206992 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:52.205442    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c6dbd69f-lrwtw: secret "image-registry-tls" not found
Oct 30 02:41:52.206992 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:52.205476    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls podName:b93ba591-108f-41bd-afed-012dcd2f8690 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:56.205463522 +0000 UTC m=+37.618919646 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls") pod "image-registry-7c6dbd69f-lrwtw" (UID: "b93ba591-108f-41bd-afed-012dcd2f8690") : secret "image-registry-tls" not found
Oct 30 02:41:52.206992 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:52.205544    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb0c5682-acc6-4029-831d-b6aa9027db25-service-ca-bundle podName:cb0c5682-acc6-4029-831d-b6aa9027db25 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:56.205533945 +0000 UTC m=+37.618990070 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cb0c5682-acc6-4029-831d-b6aa9027db25-service-ca-bundle") pod "router-default-5d4bdb4f7f-89p4x" (UID: "cb0c5682-acc6-4029-831d-b6aa9027db25") : configmap references non-existent config key: service-ca.crt
Oct 30 02:41:52.491833 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:52.491753    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-njw6d" event={"ID":"7e8de64a-fd25-4aa6-9160-b25e09d116aa","Type":"ContainerStarted","Data":"120b04ff1e94531ab5bcb4a8468cad8e3452992024094ce7f1ef4077ba45b6a8"}
Oct 30 02:41:52.532209 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:52.532112    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-njw6d" podStartSLOduration=455.172196454 podStartE2EDuration="7m37.532094673s" totalImagesPullingTime="2.359898219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:34:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:52.531762505 +0000 UTC m=+33.945218659" watchObservedRunningTime="2026-10-30 02:41:52.532094673 +0000 UTC m=+33.945550808"
Oct 30 02:41:52.652995 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:52.652918    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:52.760789 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:52.760759    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:52.880502 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:52.880475    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:52.912919 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:52.912074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:41:52.912919 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:52.912240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:52.913539 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:52.913262    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:41:52.913539 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:52.913328    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs podName:60924e48-a5e3-4f2b-ab35-48479c3b730b nodeName:}" failed. No retries permitted until 2026-10-30 02:42:24.913310034 +0000 UTC m=+66.326766159 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs") pod "network-metrics-daemon-t5747" (UID: "60924e48-a5e3-4f2b-ab35-48479c3b730b") : secret "metrics-daemon-secret" not found
Oct 30 02:41:52.917401 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:52.917350    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:53.013058 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:53.012981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwzt\" (UniqueName: \"kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt\") pod \"network-check-target-wx7bz\" (UID: \"d3d2494a-0ef0-4da6-b6dd-225f750bb909\") " pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:53.017612 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:53.017585    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:41:53.024148 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:53.024112    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkwzt\" (UniqueName: \"kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt\") pod \"network-check-target-wx7bz\" (UID: \"d3d2494a-0ef0-4da6-b6dd-225f750bb909\") " pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:53.026709 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:53.026632    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:53.355203 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:53.355146    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wx7bz"]
Oct 30 02:41:53.360258 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:53.359461    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d2494a_0ef0_4da6_b6dd_225f750bb909.slice/crio-ba00258135664cc71290793d776279e6bbefbfbfd7c99a5bcdc880a155ba2c7a WatchSource:0}: Error finding container ba00258135664cc71290793d776279e6bbefbfbfd7c99a5bcdc880a155ba2c7a: Status 404 returned error can't find the container with id ba00258135664cc71290793d776279e6bbefbfbfd7c99a5bcdc880a155ba2c7a
Oct 30 02:41:53.365011 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:53.364987    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-skc8q"]
Oct 30 02:41:53.375499 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:53.375296    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3ebba8_b807_4256_b552_5a18ec8412cb.slice/crio-c7f106bdfbea9cabe563d38be39369a6d72fb0bf702cfc49e8d00ffd79438c69 WatchSource:0}: Error finding container c7f106bdfbea9cabe563d38be39369a6d72fb0bf702cfc49e8d00ffd79438c69: Status 404 returned error can't find the container with id c7f106bdfbea9cabe563d38be39369a6d72fb0bf702cfc49e8d00ffd79438c69
Oct 30 02:41:53.503384 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:53.503303    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wx7bz" event={"ID":"d3d2494a-0ef0-4da6-b6dd-225f750bb909","Type":"ContainerStarted","Data":"ba00258135664cc71290793d776279e6bbefbfbfd7c99a5bcdc880a155ba2c7a"}
Oct 30 02:41:53.504895 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:53.504864    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-skc8q" event={"ID":"8a3ebba8-b807-4256-b552-5a18ec8412cb","Type":"ContainerStarted","Data":"c7f106bdfbea9cabe563d38be39369a6d72fb0bf702cfc49e8d00ffd79438c69"}
Oct 30 02:41:53.507533 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:53.507511    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/0.log"
Oct 30 02:41:53.507652 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:53.507548    2578 generic.go:356] "Generic (PLEG): container finished" podID="fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e" containerID="dc2f24b6f1fe134d23bcef63fb0f63e766ab51a1c805ded7810bcb48582f8ff5" exitCode=255
Oct 30 02:41:53.507652 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:53.507614    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg" event={"ID":"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e","Type":"ContainerDied","Data":"dc2f24b6f1fe134d23bcef63fb0f63e766ab51a1c805ded7810bcb48582f8ff5"}
Oct 30 02:41:53.507997 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:53.507964    2578 scope.go:122] "RemoveContainer" containerID="dc2f24b6f1fe134d23bcef63fb0f63e766ab51a1c805ded7810bcb48582f8ff5"
Oct 30 02:41:53.514268 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:53.514239    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-gxb72" event={"ID":"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9","Type":"ContainerStarted","Data":"a3a54794c55b68f6001cc496d8d6f6debe5488ea5b0320406d11685bf112882b"}
Oct 30 02:41:53.594419 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:53.594361    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-gxb72" podStartSLOduration=454.341676673 podStartE2EDuration="7m38.594341559s" totalImagesPullingTime="4.252664886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:34:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:53.592590447 +0000 UTC m=+35.006046602" watchObservedRunningTime="2026-10-30 02:41:53.594341559 +0000 UTC m=+35.007797693"
Oct 30 02:41:54.144701 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.144675    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dkfnj"]
Oct 30 02:41:54.151082 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.151055    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:54.168035 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.167896    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dkfnj"]
Oct 30 02:41:54.168035 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.168007    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:54.184247 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.184226    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 02:41:54.184330 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.184284    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 02:41:54.193756 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.193723    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8z95j\""
Oct 30 02:41:54.261830 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.261789    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:54.326875 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.326805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:54.327017 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.326872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-data-volume\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:54.327017 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.326963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:54.327116 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.327016    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-crio-socket\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:54.327116 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.327052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs9h4\" (UniqueName: \"kubernetes.io/projected/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-api-access-bs9h4\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:54.427771 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.427735    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:54.428222 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.427800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-data-volume\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:54.428222 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.427874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:54.428222 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.427929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-crio-socket\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:54.428222 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.427965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs9h4\" (UniqueName: \"kubernetes.io/projected/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-api-access-bs9h4\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:54.428419 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:54.428339    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:41:54.428419 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:54.428403    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm podName:6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da nodeName:}" failed. No retries permitted until 2026-10-30 02:41:54.92838356 +0000 UTC m=+36.341839684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dkfnj" (UID: "6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da") : configmap "kube-rbac-proxy" not found
Oct 30 02:41:54.428694 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.428675    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-data-volume\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:54.428795 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:54.428781    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:41:54.428846 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:54.428826    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls podName:6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da nodeName:}" failed. No retries permitted until 2026-10-30 02:41:54.928813518 +0000 UTC m=+36.342269642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dkfnj" (UID: "6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:41:54.429015 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.428988    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-crio-socket\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:54.455918 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.455869    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs9h4\" (UniqueName: \"kubernetes.io/projected/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-api-access-bs9h4\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:54.534307 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.534229    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg" event={"ID":"dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8","Type":"ContainerStarted","Data":"40f41f00c8f15be84f03f06c759fd7ae3540edc20469164b67bf278dba9a2e0e"}
Oct 30 02:41:54.538390 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.538298    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/1.log"
Oct 30 02:41:54.538804 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.538783    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/0.log"
Oct 30 02:41:54.538894 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.538821    2578 generic.go:356] "Generic (PLEG): container finished" podID="fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e" containerID="d49244d3852bbacd675ebd114cf975faac54dc9d1ec0822431ad05f17f237c73" exitCode=255
Oct 30 02:41:54.538894 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.538884    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg" event={"ID":"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e","Type":"ContainerDied","Data":"d49244d3852bbacd675ebd114cf975faac54dc9d1ec0822431ad05f17f237c73"}
Oct 30 02:41:54.538969 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.538918    2578 scope.go:122] "RemoveContainer" containerID="dc2f24b6f1fe134d23bcef63fb0f63e766ab51a1c805ded7810bcb48582f8ff5"
Oct 30 02:41:54.539162 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.539149    2578 scope.go:122] "RemoveContainer" containerID="d49244d3852bbacd675ebd114cf975faac54dc9d1ec0822431ad05f17f237c73"
Oct 30 02:41:54.539371 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:54.539336    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-scstg_openshift-console-operator(fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg" podUID="fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e"
Oct 30 02:41:54.544297 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.544275    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s" event={"ID":"88d453b9-7e71-4561-8e1b-4b4074651675","Type":"ContainerStarted","Data":"942369d20c6232a75314f6e6ebe1d5787995fb081871a51c5ec041bc2e7cae50"}
Oct 30 02:41:54.577929 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.577028    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg" podStartSLOduration=454.677427872 podStartE2EDuration="7m39.577011943s" totalImagesPullingTime="4.899584071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:34:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:54.575920634 +0000 UTC m=+35.989376768" watchObservedRunningTime="2026-10-30 02:41:54.577011943 +0000 UTC m=+35.990468076"
Oct 30 02:41:54.621027 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.620348    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s" podStartSLOduration=454.726052557 podStartE2EDuration="7m39.620317189s" totalImagesPullingTime="4.894264632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:34:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:54.619076324 +0000 UTC m=+36.032532456" watchObservedRunningTime="2026-10-30 02:41:54.620317189 +0000 UTC m=+36.033773323"
Oct 30 02:41:54.940675 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.940595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:54.940835 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:54.940754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:54.940835 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:54.940790    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:41:54.940942 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:54.940847    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:41:54.940942 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:54.940875    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls podName:6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da nodeName:}" failed. No retries permitted until 2026-10-30 02:41:55.940857253 +0000 UTC m=+37.354313384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dkfnj" (UID: "6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:41:54.940942 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:54.940897    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm podName:6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da nodeName:}" failed. No retries permitted until 2026-10-30 02:41:55.940883755 +0000 UTC m=+37.354339891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dkfnj" (UID: "6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da") : configmap "kube-rbac-proxy" not found
Oct 30 02:41:55.039572 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.039521    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:55.177614 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.177531    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-dgtzw"]
Oct 30 02:41:55.201038 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.200949    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:55.209938 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.209258    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-dgtzw"]
Oct 30 02:41:55.211623 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.210395    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dgtzw"
Oct 30 02:41:55.298340 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.298312    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 02:41:55.298489 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.298439    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 02:41:55.298565 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.298551    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tbftn\""
Oct 30 02:41:55.345268 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.345240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm4tl\" (UniqueName: \"kubernetes.io/projected/cd6d2859-2141-401c-825d-b6f12bb72644-kube-api-access-tm4tl\") pod \"migrator-66df5d496b-dgtzw\" (UID: \"cd6d2859-2141-401c-825d-b6f12bb72644\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dgtzw"
Oct 30 02:41:55.355454 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.355394    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:55.446759 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.446723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm4tl\" (UniqueName: \"kubernetes.io/projected/cd6d2859-2141-401c-825d-b6f12bb72644-kube-api-access-tm4tl\") pod \"migrator-66df5d496b-dgtzw\" (UID: \"cd6d2859-2141-401c-825d-b6f12bb72644\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dgtzw"
Oct 30 02:41:55.472711 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.472612    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm4tl\" (UniqueName: \"kubernetes.io/projected/cd6d2859-2141-401c-825d-b6f12bb72644-kube-api-access-tm4tl\") pod \"migrator-66df5d496b-dgtzw\" (UID: \"cd6d2859-2141-401c-825d-b6f12bb72644\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dgtzw"
Oct 30 02:41:55.481252 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.481224    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:55.544645 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.544626    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dgtzw"
Oct 30 02:41:55.549143 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.549125    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/1.log"
Oct 30 02:41:55.549485 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.549460    2578 scope.go:122] "RemoveContainer" containerID="d49244d3852bbacd675ebd114cf975faac54dc9d1ec0822431ad05f17f237c73"
Oct 30 02:41:55.549664 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:55.549644    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-scstg_openshift-console-operator(fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg" podUID="fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e"
Oct 30 02:41:55.551792 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.551770    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fx6bk" event={"ID":"f697930c-fefc-42f2-835e-7b41cf115aaf","Type":"ContainerStarted","Data":"5ac684b2576d434f29f905a7802b0d1134f60e57ec7a2de1634cdb684acf4747"}
Oct 30 02:41:55.557103 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.557080    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hpk4" event={"ID":"dfa424b6-593b-44ac-8e28-c9ba0e3207e8","Type":"ContainerStarted","Data":"35a4f2f69c650e1517ccbad327f4a4334f5e1645bf6d3da8c47e5bd1e97896d1"}
Oct 30 02:41:55.638792 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.637865    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fx6bk" podStartSLOduration=2.400979961 podStartE2EDuration="7.637845809s" totalImagesPullingTime="5.236865848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:55.636733352 +0000 UTC m=+37.050189482" watchObservedRunningTime="2026-10-30 02:41:55.637845809 +0000 UTC m=+37.051301945"
Oct 30 02:41:55.680057 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.678396    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hpk4" podStartSLOduration=451.751999579 podStartE2EDuration="7m37.678377545s" totalImagesPullingTime="5.926377966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:55.677817264 +0000 UTC m=+37.091273397" watchObservedRunningTime="2026-10-30 02:41:55.678377545 +0000 UTC m=+37.091833678"
Oct 30 02:41:55.734017 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.733944    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:55.839224 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.839159    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:55.873543 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.872324    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-dgtzw"]
Oct 30 02:41:55.873931 ip-10-0-128-241 kubenswrapper[2578]: W1030 02:41:55.873901    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6d2859_2141_401c_825d_b6f12bb72644.slice/crio-56d5467708c3d3bc9584847c948764546d4391ddc890c1cb4989e7cb17aa05e5 WatchSource:0}: Error finding container 56d5467708c3d3bc9584847c948764546d4391ddc890c1cb4989e7cb17aa05e5: Status 404 returned error can't find the container with id 56d5467708c3d3bc9584847c948764546d4391ddc890c1cb4989e7cb17aa05e5
Oct 30 02:41:55.956885 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.952201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:55.956885 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:55.952385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:55.956885 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:55.952402    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:41:55.956885 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:55.952473    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm podName:6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da nodeName:}" failed. No retries permitted until 2026-10-30 02:41:57.952455021 +0000 UTC m=+39.365911134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dkfnj" (UID: "6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da") : configmap "kube-rbac-proxy" not found
Oct 30 02:41:55.956885 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:55.952540    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:41:55.956885 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:55.952587    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls podName:6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da nodeName:}" failed. No retries permitted until 2026-10-30 02:41:57.952568671 +0000 UTC m=+39.366024799 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dkfnj" (UID: "6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:41:56.255979 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:56.255948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:41:56.256163 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:56.256006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:41:56.256163 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:56.256036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:56.256163 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:56.256062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:41:56.256163 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:56.256071    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:41:56.256163 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:56.256087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:41:56.256163 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:56.256115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb0c5682-acc6-4029-831d-b6aa9027db25-service-ca-bundle\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:41:56.256163 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:56.256145    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls podName:06715a40-fb77-4b9c-89e1-0aefe7b12a72 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:04.256124309 +0000 UTC m=+45.669580440 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7mfqp" (UID: "06715a40-fb77-4b9c-89e1-0aefe7b12a72") : secret "samples-operator-tls" not found
Oct 30 02:41:56.256805 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:56.256167    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:41:56.256805 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:56.256206    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:41:56.256805 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:56.256212    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb0c5682-acc6-4029-831d-b6aa9027db25-service-ca-bundle podName:cb0c5682-acc6-4029-831d-b6aa9027db25 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:04.256196183 +0000 UTC m=+45.669652306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cb0c5682-acc6-4029-831d-b6aa9027db25-service-ca-bundle") pod "router-default-5d4bdb4f7f-89p4x" (UID: "cb0c5682-acc6-4029-831d-b6aa9027db25") : configmap references non-existent config key: service-ca.crt
Oct 30 02:41:56.256805 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:56.256170    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:41:56.256805 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:56.256247    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls podName:d6516df6-4df4-4213-890a-0b25e274e606 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:04.256231351 +0000 UTC m=+45.669687490 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vjvcf" (UID: "d6516df6-4df4-4213-890a-0b25e274e606") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:41:56.256805 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:56.256265    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls podName:c5d42a36-81a2-49ad-9fa0-32d42e87d75d nodeName:}" failed. No retries permitted until 2026-10-30 02:42:04.256255352 +0000 UTC m=+45.669711462 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls") pod "dns-default-7mhtz" (UID: "c5d42a36-81a2-49ad-9fa0-32d42e87d75d") : secret "dns-default-metrics-tls" not found
Oct 30 02:41:56.256805 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:56.256288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jwzx9\" (UID: \"70f081e1-7202-4b86-930d-8c3f72a3bb7e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:41:56.256805 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:56.256311    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:41:56.256805 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:56.256330    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c6dbd69f-lrwtw: secret "image-registry-tls" not found
Oct 30 02:41:56.256805 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:56.256333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:41:56.256805 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:56.256364    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:41:56.256805 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:56.256380    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls podName:b93ba591-108f-41bd-afed-012dcd2f8690 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:04.256364127 +0000 UTC m=+45.669820239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls") pod "image-registry-7c6dbd69f-lrwtw" (UID: "b93ba591-108f-41bd-afed-012dcd2f8690") : secret "image-registry-tls" not found
Oct 30 02:41:56.256805 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:56.256401    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert podName:70f081e1-7202-4b86-930d-8c3f72a3bb7e nodeName:}" failed. No retries permitted until 2026-10-30 02:42:04.256390969 +0000 UTC m=+45.669847120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jwzx9" (UID: "70f081e1-7202-4b86-930d-8c3f72a3bb7e") : secret "networking-console-plugin-cert" not found
Oct 30 02:41:56.256805 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:56.256413    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs podName:cb0c5682-acc6-4029-831d-b6aa9027db25 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:04.256407556 +0000 UTC m=+45.669863666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs") pod "router-default-5d4bdb4f7f-89p4x" (UID: "cb0c5682-acc6-4029-831d-b6aa9027db25") : secret "router-metrics-certs-default" not found
Oct 30 02:41:56.256805 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:56.256413    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:41:56.256805 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:56.256424    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59bd5b5c6b-lcbtn: secret "image-registry-tls" not found
Oct 30 02:41:56.257787 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:56.256452    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls podName:bedee86a-79f7-43cf-9ed2-5dbf5b8109eb nodeName:}" failed. No retries permitted until 2026-10-30 02:42:04.256445229 +0000 UTC m=+45.669901339 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls") pod "image-registry-59bd5b5c6b-lcbtn" (UID: "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb") : secret "image-registry-tls" not found
Oct 30 02:41:56.562662 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:56.562467    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dgtzw" event={"ID":"cd6d2859-2141-401c-825d-b6f12bb72644","Type":"ContainerStarted","Data":"56d5467708c3d3bc9584847c948764546d4391ddc890c1cb4989e7cb17aa05e5"}
Oct 30 02:41:56.585978 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:56.585944    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:57.567692 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:57.567657    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wx7bz" event={"ID":"d3d2494a-0ef0-4da6-b6dd-225f750bb909","Type":"ContainerStarted","Data":"aa14ee8f4de31dd39f5899d5062149cad9313b4124199df44f7bdb550af64dfd"}
Oct 30 02:41:57.568066 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:57.567721    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:41:57.569514 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:57.569491    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dgtzw" event={"ID":"cd6d2859-2141-401c-825d-b6f12bb72644","Type":"ContainerStarted","Data":"f55314d772f644a4606e1793bd7705d37c39b8acda67fb622f4190c4183148b8"}
Oct 30 02:41:57.610575 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:57.610527    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wx7bz" podStartSLOduration=34.492344643 podStartE2EDuration="37.610513451s" totalImagesPullingTime="3.118168808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:57.609193656 +0000 UTC m=+39.022649786" watchObservedRunningTime="2026-10-30 02:41:57.610513451 +0000 UTC m=+39.023969582"
Oct 30 02:41:57.972352 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:57.972317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:57.972541 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:57.972449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:41:57.972541 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:57.972479    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:41:57.972662 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:57.972551    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:41:57.972662 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:57.972558    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls podName:6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da nodeName:}" failed. No retries permitted until 2026-10-30 02:42:01.972538942 +0000 UTC m=+43.385995067 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dkfnj" (UID: "6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:41:57.972662 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:57.972600    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm podName:6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da nodeName:}" failed. No retries permitted until 2026-10-30 02:42:01.972589095 +0000 UTC m=+43.386045206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dkfnj" (UID: "6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da") : configmap "kube-rbac-proxy" not found
Oct 30 02:41:58.613993 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:58.613946    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dgtzw" podStartSLOduration=2.024911663 podStartE2EDuration="3.613932476s" totalImagesPullingTime="1.589020813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:58.613466977 +0000 UTC m=+40.026923128" watchObservedRunningTime="2026-10-30 02:41:58.613932476 +0000 UTC m=+40.027388607"
Oct 30 02:41:58.836776 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:58.836741    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:41:58.836934 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:58.836793    2578 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:41:58.837110 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:58.837098    2578 scope.go:122] "RemoveContainer" containerID="d49244d3852bbacd675ebd114cf975faac54dc9d1ec0822431ad05f17f237c73"
Oct 30 02:41:58.837312 ip-10-0-128-241 kubenswrapper[2578]: E1030 02:41:58.837293    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-scstg_openshift-console-operator(fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg" podUID="fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e"
Oct 30 02:41:58.946490 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:58.946428    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:59.068618 ip-10-0-128-241 kubenswrapper[2578]: I1030 02:41:59.068594    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:59.279514 ip-10-0-128-241 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 02:41:59.305624 ip-10-0-128-241 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 02:41:59.305799 ip-10-0-128-241 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 02:41:59.305973 ip-10-0-128-241 systemd[1]: kubelet.service: Consumed 3.012s CPU time.
Oct 30 02:41:59.319855 ip-10-0-128-241 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:41:59.413037 ip-10-0-128-241 kubenswrapper[4960]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:41:59.413037 ip-10-0-128-241 kubenswrapper[4960]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:41:59.413037 ip-10-0-128-241 kubenswrapper[4960]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:41:59.413037 ip-10-0-128-241 kubenswrapper[4960]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:41:59.415149 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415078    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:59.415149 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415150    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:59.415149 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415154    4960 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:59.415278 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415157    4960 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:59.415278 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415160    4960 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:59.415278 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415163    4960 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:59.415278 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415166    4960 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:59.415278 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415169    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:59.415278 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415171    4960 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:59.415278 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415174    4960 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:59.415278 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415191    4960 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:59.415278 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415195    4960 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:59.415278 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415198    4960 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:59.415278 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415200    4960 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:59.415278 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415204    4960 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:59.415278 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415208    4960 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:59.415278 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415212    4960 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:59.415278 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415215    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:59.415278 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415218    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:59.415278 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415221    4960 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:59.415278 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415224    4960 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:59.415278 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415232    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:59.415735 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415235    4960 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:59.415735 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415238    4960 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:59.415735 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415240    4960 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:59.415735 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415243    4960 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:59.415735 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415246    4960 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:59.415735 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415249    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:59.415735 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415251    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:59.415735 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415254    4960 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:59.415735 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415257    4960 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:59.415735 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415260    4960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:59.415735 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415262    4960 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:59.415735 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415265    4960 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:59.415735 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415268    4960 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:59.415735 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415270    4960 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:59.415735 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415274    4960 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:59.415735 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415277    4960 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:59.415735 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415280    4960 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:59.415735 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415283    4960 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:59.415735 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415286    4960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:59.416203 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415289    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:59.416203 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415292    4960 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:59.416203 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415294    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:59.416203 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415297    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:59.416203 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415300    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:59.416203 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415302    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:59.416203 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415305    4960 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:59.416203 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415308    4960 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:59.416203 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415311    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:59.416203 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415314    4960 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:59.416203 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415317    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:59.416203 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415319    4960 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:59.416203 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415322    4960 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:59.416203 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415325    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:59.416203 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415328    4960 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:59.416203 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415331    4960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:59.416203 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415334    4960 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:59.416203 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415337    4960 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:59.416203 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415339    4960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415342    4960 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415346    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415348    4960 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415351    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415354    4960 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415356    4960 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415359    4960 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415362    4960 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415365    4960 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415367    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415370    4960 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415373    4960 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415376    4960 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415379    4960 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415382    4960 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415384    4960 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415387    4960 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415391    4960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415394    4960 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:59.416666 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415396    4960 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415399    4960 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415402    4960 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415404    4960 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415407    4960 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415410    4960 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415412    4960 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415415    4960 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415417    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415420    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415423    4960 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415426    4960 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415429    4960 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415431    4960 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415434    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415437    4960 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415439    4960 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415444    4960 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415448    4960 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415451    4960 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:59.417159 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415455    4960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:59.417651 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415458    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:59.417651 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415461    4960 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:59.417651 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415465    4960 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:59.417651 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415468    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:59.417651 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415474    4960 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:59.417651 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.415477    4960 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:59.417651 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416201    4960 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:59.417651 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416208    4960 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:59.417651 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416211    4960 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:59.417651 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416214    4960 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:59.417651 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416217    4960 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:59.417651 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416220    4960 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:59.417651 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416224    4960 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:59.417651 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416228    4960 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:59.417651 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416231    4960 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:59.417651 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416234    4960 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:59.417651 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416237    4960 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:59.417651 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416240    4960 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:59.417651 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416242    4960 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:59.418148 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416246    4960 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:59.418148 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416248    4960 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:59.418148 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416251    4960 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:59.418148 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416254    4960 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:59.418148 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416256    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:59.418148 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416259    4960 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:59.418148 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416262    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:59.418148 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416265    4960 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:59.418148 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416267    4960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:59.418148 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416270    4960 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:59.418148 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416273    4960 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:59.418148 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416275    4960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:59.418148 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416278    4960 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:59.418148 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416280    4960 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:59.418148 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416283    4960 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:59.418148 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416285    4960 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:59.418148 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416288    4960 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:59.418148 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416293    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:59.418148 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416296    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416299    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416302    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416305    4960 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416308    4960 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416310    4960 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416313    4960 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416316    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416319    4960 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416321    4960 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416324    4960 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416327    4960 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416329    4960 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416332    4960 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416335    4960 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416337    4960 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416340    4960 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416343    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416345    4960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416348    4960 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:59.418637 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416351    4960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:59.419259 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416354    4960 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:59.419259 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416356    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:59.419259 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416359    4960 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:59.419259 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416361    4960 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:59.419259 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416364    4960 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:59.419259 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416366    4960 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:59.419259 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416369    4960 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:59.419259 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416372    4960 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:59.419259 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416374    4960 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:59.419259 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416377    4960 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:59.419259 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416381    4960 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:59.419259 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416384    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:59.419259 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416386    4960 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:59.419259 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416389    4960 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:59.419259 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416393    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:59.419259 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416395    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:59.419259 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416398    4960 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:59.419259 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416403    4960 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:59.419259 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416406    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:59.419726 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416408    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:59.419726 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416411    4960 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:59.419726 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416414    4960 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:59.419726 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416416    4960 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:59.419726 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416419    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:59.419726 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416422    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:59.419726 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416424    4960 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:59.419726 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416427    4960 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:59.419726 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416432    4960 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:59.419726 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416436    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:59.419726 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416440    4960 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:59.419726 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416443    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:59.419726 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416445    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:59.419726 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416448    4960 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:59.419726 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416451    4960 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:59.419726 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416453    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:59.419726 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416456    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:59.419726 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416459    4960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:59.419726 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416461    4960 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416464    4960 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416467    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416470    4960 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416473    4960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416477    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416479    4960 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416482    4960 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416485    4960 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416488    4960 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416490    4960 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416493    4960 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416497    4960 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416500    4960 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416502    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416505    4960 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416509    4960 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416982    4960 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416988    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416992    4960 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:59.420196 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416994    4960 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:59.420709 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.416997    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:59.420709 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417000    4960 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:59.420709 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417003    4960 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:59.420709 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417005    4960 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:59.420709 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417008    4960 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:59.420709 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417011    4960 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:59.420709 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417013    4960 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:59.420709 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417016    4960 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:59.420709 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417019    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:59.420709 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417021    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:59.420709 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417024    4960 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:59.420709 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417027    4960 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:59.420709 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417030    4960 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:59.420709 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417032    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:59.420709 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417036    4960 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:59.420709 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417041    4960 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:59.420709 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417046    4960 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:59.420709 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417049    4960 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:59.420709 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417052    4960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417054    4960 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417057    4960 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417060    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417063    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417067    4960 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417069    4960 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417072    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417074    4960 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417077    4960 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417080    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417082    4960 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417085    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417087    4960 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417090    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417092    4960 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417095    4960 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417097    4960 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417100    4960 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417103    4960 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:59.421198 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417105    4960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:59.421682 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417108    4960 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:59.421682 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417111    4960 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:59.421682 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417115    4960 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:59.421682 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417118    4960 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:59.421682 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417121    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:59.421682 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417124    4960 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:59.421682 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417127    4960 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:59.421682 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417130    4960 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:59.421682 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417132    4960 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:59.421682 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417136    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:59.421682 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417139    4960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:59.421682 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417141    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:59.421682 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417144    4960 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:59.421682 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417147    4960 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:59.421682 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417150    4960 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:59.421682 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417153    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:59.421682 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417155    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:59.421682 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417159    4960 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:59.421682 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417161    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:59.422163 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417164    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:59.422163 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417167    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:59.422163 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417170    4960 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:59.422163 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417172    4960 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:59.422163 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417189    4960 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:59.422163 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417192    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:59.422163 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417195    4960 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:59.422163 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417197    4960 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:59.422163 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417200    4960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:59.422163 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417203    4960 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:59.422163 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417205    4960 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:59.422163 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417208    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:59.422163 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417211    4960 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:59.422163 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417213    4960 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:59.422163 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417216    4960 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:59.422163 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417219    4960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:59.422163 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417222    4960 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:59.422163 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417225    4960 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:59.422163 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417228    4960 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:59.422771 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417230    4960 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:59.422771 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417233    4960 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:59.422771 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417236    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:59.422771 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417240    4960 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:59.422771 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417243    4960 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:59.422771 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417246    4960 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:59.422771 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417248    4960 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:59.422771 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417251    4960 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:59.422771 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417254    4960 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:59.422771 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417257    4960 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:59.422771 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417259    4960 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:59.422771 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417262    4960 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:59.422771 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417265    4960 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:59.422771 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417267    4960 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:59.422771 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417270    4960 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:59.422771 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417272    4960 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:59.422771 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417275    4960 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:59.422771 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417278    4960 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:59.422771 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417281    4960 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417283    4960 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417286    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417289    4960 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417291    4960 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417294    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.417296    4960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417385    4960 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417396    4960 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417402    4960 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417406    4960 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417410    4960 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417413    4960 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417417    4960 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417421    4960 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417424    4960 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417427    4960 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417433    4960 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417436    4960 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417439    4960 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417442    4960 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417445    4960 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417448    4960 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:41:59.423537 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417451    4960 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417455    4960 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417459    4960 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417462    4960 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417466    4960 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417469    4960 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417472    4960 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417476    4960 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417479    4960 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417482    4960 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417486    4960 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417489    4960 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417492    4960 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417495    4960 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417499    4960 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417503    4960 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417507    4960 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417510    4960 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417513    4960 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417516    4960 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417519    4960 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417522    4960 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417526    4960 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417529    4960 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:41:59.424087 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417532    4960 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417535    4960 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417540    4960 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417544    4960 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417547    4960 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417550    4960 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417554    4960 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417557    4960 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417560    4960 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417563    4960 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417566    4960 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417569    4960 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417572    4960 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417575    4960 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417579    4960 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417582    4960 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417585    4960 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417589    4960 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417592    4960 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417595    4960 flags.go:64] FLAG: --help="false"
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417598    4960 flags.go:64] FLAG: --hostname-override="ip-10-0-128-241.ec2.internal"
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417601    4960 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417605    4960 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:41:59.424720 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417609    4960 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417613    4960 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417616    4960 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417619    4960 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417622    4960 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417625    4960 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417628    4960 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417632    4960 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417635    4960 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417638    4960 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417641    4960 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417645    4960 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417648    4960 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417651    4960 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417654    4960 flags.go:64] FLAG: --lock-file=""
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417657    4960 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417660    4960 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417664    4960 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417669    4960 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417672    4960 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417676    4960 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417679    4960 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417682    4960 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417685    4960 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:41:59.425312 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417688    4960 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417691    4960 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417695    4960 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417698    4960 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417702    4960 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417705    4960 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417708    4960 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417714    4960 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417717    4960 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417720    4960 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417723    4960 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417726    4960 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417733    4960 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417736    4960 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417739    4960 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417742    4960 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417745    4960 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417748    4960 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417751    4960 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417755    4960 flags.go:64] FLAG: --port="10250"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417758    4960 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417761    4960 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ecaa8c911d2983f9"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417764    4960 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417767    4960 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417770    4960 flags.go:64] FLAG: --register-node="true"
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417773    4960 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:41:59.425867 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417777    4960 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417780    4960 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417783    4960 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417786    4960 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417790    4960 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417793    4960 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417796    4960 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417799    4960 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417802    4960 flags.go:64] FLAG: --runonce="false"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417805    4960 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417808    4960 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417811    4960 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417814    4960 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417818    4960 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417822    4960 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417825    4960 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417828    4960 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417831    4960 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417834    4960 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417837    4960 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417840    4960 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417843    4960 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417846    4960 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417849    4960 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417854    4960 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:41:59.426494 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417857    4960 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417860    4960 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417865    4960 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417867    4960 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417870    4960 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417873    4960 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417876    4960 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417879    4960 flags.go:64] FLAG: --v="2"
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417884    4960 flags.go:64] FLAG: --version="false"
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417888    4960 flags.go:64] FLAG: --vmodule=""
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417892    4960 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.417895    4960 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418020    4960 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418024    4960 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418027    4960 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418030    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418033    4960 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418036    4960 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418039    4960 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418042    4960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418045    4960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418048    4960 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418051    4960 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:59.427140 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418053    4960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:59.427711 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418056    4960 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:59.427711 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418059    4960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:59.427711 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418061    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:59.427711 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418064    4960 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:59.427711 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418067    4960 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:59.427711 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418070    4960 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:59.427711 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418072    4960 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:59.427711 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418075    4960 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:59.427711 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418078    4960 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:59.427711 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418080    4960 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:59.427711 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418083    4960 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:59.427711 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418086    4960 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:59.427711 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418088    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:59.427711 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418091    4960 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:59.427711 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418093    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:59.427711 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418096    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:59.427711 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418099    4960 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:59.427711 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418102    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:59.427711 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418104    4960 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418107    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418110    4960 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418113    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418115    4960 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418118    4960 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418120    4960 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418123    4960 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418126    4960 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418128    4960 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418133    4960 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418136    4960 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418139    4960 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418141    4960 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418144    4960 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418147    4960 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418149    4960 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418152    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418155    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418158    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:59.428415 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418160    4960 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:59.429029 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418163    4960 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:59.429029 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418166    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:59.429029 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418168    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:59.429029 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418173    4960 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:59.429029 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418190    4960 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:59.429029 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418195    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:59.429029 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418198    4960 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:59.429029 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418201    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:59.429029 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418204    4960 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:59.429029 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418206    4960 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:59.429029 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418209    4960 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:59.429029 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418212    4960 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:59.429029 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418215    4960 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:59.429029 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418219    4960 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:59.429029 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418223    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:59.429029 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418226    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:59.429029 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418229    4960 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:59.429029 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418232    4960 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:59.429029 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418235    4960 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418238    4960 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418240    4960 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418245    4960 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418248    4960 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418250    4960 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418253    4960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418256    4960 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418259    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418262    4960 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418264    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418267    4960 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418270    4960 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418273    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418275    4960 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418278    4960 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418281    4960 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418283    4960 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418286    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418290    4960 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:59.429724 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418293    4960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:59.430337 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418296    4960 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:59.430337 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418299    4960 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:59.430337 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418301    4960 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:59.430337 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418304    4960 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:59.430337 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418307    4960 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:59.430337 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418310    4960 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:59.430337 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418313    4960 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:59.430337 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418315    4960 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:59.430337 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418319    4960 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:59.430337 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418322    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:59.430337 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418325    4960 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:59.430337 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418327    4960 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:59.430337 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418330    4960 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:59.430337 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418332    4960 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:59.430337 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418336    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:59.430337 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418339    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:59.430728 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.418365    4960 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:41:59.430728 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418400    4960 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:59.430728 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418404    4960 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:59.430728 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418407    4960 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:59.430728 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418410    4960 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:59.430728 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418413    4960 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:59.430728 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418416    4960 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:59.430728 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418418    4960 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:59.430728 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418421    4960 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:59.430728 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418424    4960 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:59.430728 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418426    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:59.430728 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418429    4960 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:59.430728 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418432    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:59.430728 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418435    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:59.430728 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418439    4960 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:59.430728 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418442    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418445    4960 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418447    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418450    4960 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418452    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418455    4960 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418458    4960 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418460    4960 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418463    4960 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418465    4960 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418469    4960 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418471    4960 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418474    4960 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418477    4960 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418480    4960 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418482    4960 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418486    4960 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418489    4960 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418491    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418494    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:59.431154 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418497    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:59.431789 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418499    4960 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:59.431789 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418502    4960 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:59.431789 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418504    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:59.431789 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418507    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:59.431789 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418511    4960 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:59.431789 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418514    4960 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:59.431789 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418518    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:59.431789 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418520    4960 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:59.431789 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418523    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:59.431789 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418525    4960 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:59.431789 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418530    4960 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:59.431789 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418532    4960 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:59.431789 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418535    4960 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:59.431789 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418537    4960 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:59.431789 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418541    4960 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:59.431789 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418545    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:59.431789 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418548    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:59.431789 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418551    4960 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418554    4960 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418557    4960 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418559    4960 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418562    4960 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418565    4960 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418570    4960 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418573    4960 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418575    4960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418578    4960 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418582    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418585    4960 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418587    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418590    4960 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418593    4960 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418595    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418598    4960 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418601    4960 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418603    4960 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418606    4960 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:59.432444 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418609    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418611    4960 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418614    4960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418616    4960 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418627    4960 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418631    4960 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418634    4960 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418638    4960 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418641    4960 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418645    4960 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418648    4960 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418650    4960 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418653    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418656    4960 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418658    4960 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418661    4960 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418664    4960 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418666    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418669    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418672    4960 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:59.432917 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418675    4960 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:59.433509 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418678    4960 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:59.433509 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418682    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:59.433509 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418684    4960 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:59.433509 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418688    4960 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:59.433509 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418691    4960 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:59.433509 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418693    4960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:59.433509 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418696    4960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:59.433509 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418699    4960 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:59.433509 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418701    4960 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:59.433509 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418704    4960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:59.433509 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418707    4960 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:59.433509 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418709    4960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:59.433509 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.418712    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:59.433509 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.423234    4960 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:41:59.433509 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.423263    4960 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:41:59.433509 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.423288    4960 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:41:59.433509 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.423295    4960 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:41:59.433509 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423777    4960 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:59.433509 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423791    4960 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:59.433978 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423798    4960 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:59.433978 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423803    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:59.433978 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423807    4960 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:59.433978 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423812    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:59.433978 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423817    4960 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:59.433978 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423822    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:59.433978 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423826    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:59.433978 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423831    4960 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:59.433978 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423835    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:59.433978 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423840    4960 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:59.433978 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423845    4960 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:59.433978 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423850    4960 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:59.433978 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423855    4960 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:59.433978 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423859    4960 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:59.433978 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423864    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:59.433978 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423868    4960 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:59.433978 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423872    4960 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:59.433978 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423878    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:59.433978 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423885    4960 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:59.434468 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423892    4960 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:59.434468 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423898    4960 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:59.434468 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423905    4960 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:59.434468 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423910    4960 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:59.434468 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423915    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:59.434468 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423920    4960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:59.434468 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423925    4960 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:59.434468 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423930    4960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:59.434468 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423935    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:59.434468 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423941    4960 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:59.434468 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423946    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:59.434468 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423951    4960 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:59.434468 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423955    4960 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:59.434468 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423960    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:59.434468 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423964    4960 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:59.434468 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423970    4960 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:59.434468 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423973    4960 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:59.434468 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423976    4960 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:59.434468 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423979    4960 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423982    4960 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423984    4960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423987    4960 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423990    4960 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423993    4960 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423995    4960 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.423998    4960 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424001    4960 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424004    4960 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424007    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424010    4960 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424013    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424016    4960 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424018    4960 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424021    4960 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424024    4960 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424026    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424029    4960 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424032    4960 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:59.435042 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424035    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:59.435661 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424038    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:59.435661 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424041    4960 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:59.435661 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424044    4960 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:59.435661 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424048    4960 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:59.435661 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424052    4960 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:59.435661 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424055    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:59.435661 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424058    4960 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:59.435661 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424061    4960 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:59.435661 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424064    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:59.435661 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424066    4960 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:59.435661 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424069    4960 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:59.435661 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424072    4960 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:59.435661 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424074    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:59.435661 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424077    4960 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:59.435661 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424080    4960 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:59.435661 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424082    4960 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:59.435661 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424085    4960 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:59.435661 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424088    4960 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:59.435661 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424090    4960 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424094    4960 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424097    4960 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424100    4960 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424102    4960 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424105    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424108    4960 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424111    4960 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424113    4960 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424116    4960 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424119    4960 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424122    4960 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424124    4960 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424127    4960 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424129    4960 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424132    4960 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424135    4960 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424137    4960 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424140    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424142    4960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:59.436305 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424145    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:59.436960 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424148    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:59.436960 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424150    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:59.436960 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424153    4960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:59.436960 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424155    4960 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:59.436960 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424158    4960 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:59.436960 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424160    4960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:59.436960 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424163    4960 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:59.436960 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.424223    4960 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:41:59.436960 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424398    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:59.436960 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424403    4960 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:59.436960 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424406    4960 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:59.436960 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424408    4960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:59.436960 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424411    4960 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:59.436960 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424415    4960 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:59.436960 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424418    4960 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:59.436960 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424421    4960 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424424    4960 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424427    4960 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424429    4960 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424432    4960 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424435    4960 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424438    4960 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424440    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424443    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424446    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424449    4960 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424451    4960 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424454    4960 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424456    4960 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424459    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424462    4960 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424465    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424468    4960 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424470    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424474    4960 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:59.437419 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424478    4960 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424481    4960 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424484    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424486    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424489    4960 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424492    4960 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424495    4960 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424498    4960 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424500    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424503    4960 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424506    4960 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424509    4960 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424512    4960 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424515    4960 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424518    4960 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424521    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424523    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424526    4960 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424529    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424532    4960 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:59.437929 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424534    4960 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:59.438427 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424538    4960 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:59.438427 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424543    4960 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:59.438427 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424546    4960 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:59.438427 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424549    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:59.438427 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424552    4960 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:59.438427 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424554    4960 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:59.438427 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424557    4960 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:59.438427 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424560    4960 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:59.438427 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424563    4960 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:59.438427 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424565    4960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:59.438427 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424568    4960 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:59.438427 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424571    4960 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:59.438427 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424573    4960 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:59.438427 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424576    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:59.438427 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424578    4960 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:59.438427 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424581    4960 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:59.438427 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424584    4960 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:59.438427 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424586    4960 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:59.438427 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424589    4960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424592    4960 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424594    4960 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424597    4960 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424600    4960 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424602    4960 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424605    4960 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424608    4960 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424610    4960 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424613    4960 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424615    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424618    4960 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424621    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424624    4960 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424626    4960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424629    4960 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424632    4960 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424634    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424637    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424639    4960 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:59.438976 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424642    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:59.439480 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424645    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:59.439480 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424647    4960 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:59.439480 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424650    4960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:59.439480 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424652    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:59.439480 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424655    4960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:59.439480 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424657    4960 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:59.439480 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424660    4960 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:59.439480 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424663    4960 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:59.439480 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424665    4960 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:59.439480 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424668    4960 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:59.439480 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424671    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:59.439480 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424673    4960 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:59.439480 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424676    4960 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:59.439480 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424678    4960 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:59.439480 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424681    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:59.439480 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424683    4960 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:59.439480 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424686    4960 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:59.439480 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424689    4960 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:59.439480 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424692    4960 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:59.439969 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.424716    4960 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:41:59.439969 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424748    4960 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:59.439969 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424753    4960 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:59.439969 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424757    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:59.439969 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424760    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:59.439969 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424763    4960 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:59.439969 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424766    4960 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:59.439969 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424769    4960 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:59.439969 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424771    4960 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:59.439969 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424774    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:59.439969 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424777    4960 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:59.439969 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424779    4960 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:59.439969 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424782    4960 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:59.439969 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424785    4960 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:59.439969 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424788    4960 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:59.439969 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424791    4960 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:59.440620 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424793    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:59.440620 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424796    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:59.440620 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424798    4960 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:59.440620 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424801    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:59.440620 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424804    4960 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:59.440620 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424806    4960 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:59.440620 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424810    4960 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:59.440620 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424814    4960 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:59.440620 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424817    4960 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:59.440620 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424820    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:59.440620 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424823    4960 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:59.440620 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424825    4960 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:59.440620 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424830    4960 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:59.440620 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424833    4960 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:59.440620 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424836    4960 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:59.440620 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424839    4960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:59.440620 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424842    4960 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:59.440620 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424845    4960 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:59.440620 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424848    4960 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424850    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424853    4960 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424856    4960 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424859    4960 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424862    4960 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424865    4960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424868    4960 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424870    4960 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424873    4960 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424876    4960 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424878    4960 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424881    4960 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424884    4960 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424886    4960 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424889    4960 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424891    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424894    4960 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424897    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424900    4960 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:59.441491 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424902    4960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424905    4960 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424907    4960 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424910    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424912    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424915    4960 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424919    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424922    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424924    4960 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424927    4960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424930    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424933    4960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424936    4960 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424939    4960 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424942    4960 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424944    4960 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424947    4960 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424949    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424952    4960 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424955    4960 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:59.442255 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424958    4960 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424960    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424963    4960 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424965    4960 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424968    4960 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424971    4960 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424973    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424976    4960 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424979    4960 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424981    4960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424984    4960 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424987    4960 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424989    4960 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424992    4960 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424994    4960 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.424997    4960 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.425000    4960 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.425002    4960 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.425006    4960 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.425009    4960 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:59.442869 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.425011    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:59.443714 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.425014    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:59.443714 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.425017    4960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:59.443714 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.425020    4960 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:59.443714 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.425022    4960 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:59.443714 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.425025    4960 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:59.443714 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.425028    4960 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:59.443714 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.425030    4960 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:59.443714 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.425033    4960 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:59.443714 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.425036    4960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:59.443714 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.425038    4960 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:59.443714 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.425041    4960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:59.443714 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:41:59.425045    4960 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:59.443714 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.425095    4960 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:41:59.443714 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.426107    4960 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 02:41:59.443714 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.426174    4960 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 02:41:59.443714 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.426391    4960 server.go:1014] "Starting client certificate rotation"
Oct 30 02:41:59.443714 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.426523    4960 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:41:59.444504 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.426568    4960 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:36:19 +0000 UTC" deadline="2028-04-15 13:50:42.698939518 +0000 UTC"
Oct 30 02:41:59.444504 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.426596    4960 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12803h8m43.272345886s"
Oct 30 02:41:59.444504 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.426883    4960 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:41:59.444504 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.427960    4960 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:41:59.444504 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.429510    4960 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:41:59.444504 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.431202    4960 log.go:25] "Validated CRI v1 image API"
Oct 30 02:41:59.444504 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.431906    4960 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:41:59.444504 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.433741    4960 fs.go:135] Filesystem UUIDs: map[0e719797-d576-4626-b058-ac75bd407120:/dev/nvme0n1p4 2bd6444e-b28d-46e0-a215-130aed9f2ded:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 30 02:41:59.444891 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.433761    4960 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02d8d231e9a19a0f506667e517f3cdf3993ea52b4b74c83b8a88a120b220d04c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02d8d231e9a19a0f506667e517f3cdf3993ea52b4b74c83b8a88a120b220d04c/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/180be8420badf19cba5a9a9e962f858c4bc29b011327502f30c53c0a27010815/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/180be8420badf19cba5a9a9e962f858c4bc29b011327502f30c53c0a27010815/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32a22c148d94150ee483058da90f16b3b35fbf12fc9824acf14aa476872056c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32a22c148d94150ee483058da90f16b3b35fbf12fc9824acf14aa476872056c4/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f83389d86901aa144c3416cb26dd8e5d69a416640db5e895259a44226353e59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f83389d86901aa144c3416cb26dd8e5d69a416640db5e895259a44226353e59/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/410a6e7ef4e47afad48737d7e85659ac5aa54e1fb2e993f98cfd49d5410a53d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/410a6e7ef4e47afad48737d7e85659ac5aa54e1fb2e993f98cfd49d5410a53d8/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/438045a0b01f1568bab7043e3ba8e47cef5fb051e72d0adb845357919c6dbd15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/438045a0b01f1568bab7043e3ba8e47cef5fb051e72d0adb845357919c6dbd15/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d98adb609a438f29cc18c1911ff35f2e0c45c42c5078807bb7927506095c6c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d98adb609a438f29cc18c1911ff35f2e0c45c42c5078807bb7927506095c6c3/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5668c9636dbb9aff7816e1059ac0e651e49fb53431a60899b0f30e2240b0c1bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5668c9636dbb9aff7816e1059ac0e651e49fb53431a60899b0f30e2240b0c1bb/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/56d5467708c3d3bc9584847c948764546d4391ddc890c1cb4989e7cb17aa05e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/56d5467708c3d3bc9584847c948764546d4391ddc890c1cb4989e7cb17aa05e5/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6218f687ee9a5ad9e74be00e02ab9145085a27d8dffebf2424985e9f4f355b79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6218f687ee9a5ad9e74be00e02ab9145085a27d8dffebf2424985e9f4f355b79/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/639d650dbcaa5a892eaaf854b9eb42ddb3692ca7e6d6e99b7ce5b9e7e9954633/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/639d650dbcaa5a892eaaf854b9eb42ddb3692ca7e6d6e99b7ce5b9e7e9954633/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66b0866bd0781a66cf486595a939b5ec710e86dc87d6021b5f57cbff2c2f2a7a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66b0866bd0781a66cf486595a939b5ec710e86dc87d6021b5f57cbff2c2f2a7a/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/67b82271ccb3ea645450dcfdbe744c68725094663336981be1d336bf4de228c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/67b82271ccb3ea645450dcfdbe744c68725094663336981be1d336bf4de228c8/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/753ec90d7ab7e4722c76de269ba3c7110ee43e363062e9fd3e563965414d2bda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/753ec90d7ab7e4722c76de269ba3c7110ee43e363062e9fd3e563965414d2bda/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91181e4e17e08fed6169aeca64410d61a4ff599dcd8c981d467db052e2d24cdb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91181e4e17e08fed6169aeca64410d61a4ff599dcd8c981d467db052e2d24cdb/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba00258135664cc71290793d776279e6bbefbfbfd7c99a5bcdc880a155ba2c7a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba00258135664cc71290793d776279e6bbefbfbfd7c99a5bcdc880a155ba2c7a/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7f106bdfbea9cabe563d38be39369a6d72fb0bf702cfc49e8d00ffd79438c69/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7f106bdfbea9cabe563d38be39369a6d72fb0bf702cfc49e8d00ffd79438c69/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2fb4e9ea19f875c664825b0926b6a616905fc4f12ef6853c726a5394985bf8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2fb4e9ea19f875c664825b0926b6a616905fc4f12ef6853c726a5394985bf8e/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e31c17097aecc157404375a3654588a06aae2b3b39f29448f0da4bae02470c79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e31c17097aecc157404375a3654588a06aae2b3b39f29448f0da4bae02470c79/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6fc49c2f7adc853e0db6bde219654ea6290fed8a8b397ef2ac48964c122a386/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6fc49c2f7adc853e0db6bde219654ea6290fed8a8b397ef2ac48964c122a386/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7a3b1ae985324333bafc677eb2ce527fe499935b2c7ab6fabc34cc6f7b16e37/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7a3b1ae985324333bafc677eb2ce527fe499935b2c7ab6fabc34cc6f7b16e37/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ecdf5dc72cb424d2f4b3cd5ed9fd3beea31f652ce5a66a5f2954ba9dfded6365/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ecdf5dc72cb424d2f4b3cd5ed9fd3beea31f652ce5a66a5f2954ba9dfded6365/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f28ccef2af82f377429f80311d72cbb000f56833b5b5376c0eda08d7be2fc126/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f28ccef2af82f377429f80311d72cbb000f56833b5b5376c0eda08d7be2fc126/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06715a40-fb77-4b9c-89e1-0aefe7b12a72/volumes/kubernetes.io~projected/kube-api-access-jxg9s:{mountpoint:/var/lib/kubelet/pods/06715a40-fb77-4b9c-89e1-0aefe7b12a72/volumes/kubernetes.io~projected/kube-api-access-jxg9s major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d2d9d6b-2705-49d3-b3d2-079d6d85148f/volumes/kubernetes.io~projected/kube-api-access-tpm79:{mountpoint:/var/lib/kubelet/pods/0d2d9d6b-2705-49d3-b3d2-079d6d85148f/volumes/kubernetes.io~projected/kube-api-access-tpm79 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38e84dd1-8ee0-4a74-8802-3e3200c96642/volumes/kubernetes.io~projected/kube-api-access-cshhd:{mountpoint:/var/lib/kubelet/pods/38e84dd1-8ee0-4a74-8802-3e3200c96642/volumes/kubernetes.io~projected/kube-api-access-cshhd major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c022b3d-db3f-45a7-b6b7-5404c6f170c8/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/3c022b3d-db3f-45a7-b6b7-5404c6f170c8/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c022b3d-db3f-45a7-b6b7-5404c6f170c8/volumes/kubernetes.io~projected/kube-api-access-sl9cq:{mountpoint:/var/lib/kubelet/pods/3c022b3d-db3f-45a7-b6b7-5404c6f170c8/volumes/kubernetes.io~projected/kube-api-access-sl9cq major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c022b3d-db3f-45a7-b6b7-5404c6f170c8/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/3c022b3d-db3f-45a7-b6b7-5404c6f170c8/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49a36e47-9ab6-49f4-bde8-85b78ba7c208/volumes/kubernetes.io~projected/kube-api-access-6sv8m:{mountpoint:/var/lib/kubelet/pods/49a36e47-9ab6-49f4-bde8-85b78ba7c208/volumes/kubernetes.io~projected/kube-api-access-6sv8m major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56c4a0de-1d41-4b83-9bac-ddee3f43879b/volumes/kubernetes.io~projected/kube-api-access-jnqw2:{mountpoint:/var/lib/kubelet/pods/56c4a0de-1d41-4b83-9bac-ddee3f43879b/volumes/kubernetes.io~projected/kube-api-access-jnqw2 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60924e48-a5e3-4f2b-ab35-48479c3b730b/volumes/kubernetes.io~projected/kube-api-access-lqbmt:{mountpoint:/var/lib/kubelet/pods/60924e48-a5e3-4f2b-ab35-48479c3b730b/volumes/kubernetes.io~projected/kube-api-access-lqbmt major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da/volumes/kubernetes.io~projected/kube-api-access-bs9h4:{mountpoint:/var/lib/kubelet/pods/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da/volumes/kubernetes.io~projected/kube-api-access-bs9h4 major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75211257-e617-40de-bf22-878d736448d1/volumes/kubernetes.io~projected/kube-api-access-lpt6g:{mountpoint:/var/lib/kubelet/pods/75211257-e617-40de-bf22-878d736448d1/volumes/kubernetes.io~projected/kube-api-access-lpt6g major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e8de64a-fd25-4aa6-9160-b25e09d116aa/volumes/kubernetes.io~projected/kube-api-access-wl6q6:{mountpoint:/var/lib/kubelet/pods/7e8de64a-fd25-4aa6-9160-b25e09d116aa/volumes/kubernetes.io~projected/kube-api-access-wl6q6 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88d453b9-7e71-4561-8e1b-4b4074651675/volumes/kubernetes.io~projected/kube-api-access-svt6f:{mountpoint:/var/lib/kubelet/pods/88d453b9-7e71-4561-8e1b-4b4074651675/volumes/kubernetes.io~projected/kube-api-access-svt6f major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88d453b9-7e71-4561-8e1b-4b4074651675/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/88d453b9-7e71-4561-8e1b-4b4074651675/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/895d8ed4-6421-4018-a630-fb995aadf0bc/volumes/kubernetes.io~projected/kube-api-access-6kh57:{mountpoint:/var/lib/kubelet/pods/895d8ed4-6421-4018-a630-fb995aadf0bc/volumes/kubernetes.io~projected/kube-api-access-6kh57 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a3ebba8-b807-4256-b552-5a18ec8412cb/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8a3ebba8-b807-4256-b552-5a18ec8412cb/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7/volumes/kubernetes.io~projected/kube-api-access-lpf67:{mountpoint:/var/lib/kubelet/pods/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7/volumes/kubernetes.io~projected/kube-api-access-lpf67 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1337da2-13ee-4b79-9a84-37ef217ae10f/volumes/kubernetes.io~projected/kube-api-access-6j6pt:{mountpoint:/var/lib/kubelet/pods/b1337da2-13ee-4b79-9a84-37ef217ae10f/volumes/kubernetes.io~projected/kube-api-access-6j6pt major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b93ba591-108f-41bd-afed-012dcd2f8690/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/b93ba591-108f-41bd-afed-012dcd2f8690/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b93ba591-108f-41bd-afed-012dcd2f8690/volumes/kubernetes.io~projected/kube-api-access-hmg6l:{mountpoint:/var/lib/kubelet/pods/b93ba591-108f-41bd-afed-012dcd2f8690/volumes/kubernetes.io~projected/kube-api-access-hmg6l major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b93ba591-108f-41bd-afed-012dcd2f8690/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/b93ba591-108f-41bd-afed-012dcd2f8690/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b93ba591-108f-41bd-afed-012dcd2f8690/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/b93ba591-108f-41bd-afed-012dcd2f8690/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb/volumes/kubernetes.io~projected/kube-api-access-dxvfq:{mountpoint:/var/lib/kubelet/pods/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb/volumes/kubernetes.io~projected/kube-api-access-dxvfq major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1cf080d-9f7d-4614-b446-76d174d9c3f7/volumes/kubernetes.io~projected/kube-api-access-xs8x4:{mountpoint:/var/lib/kubelet/pods/c1cf080d-9f7d-4614-b446-76d174d9c3f7/volumes/kubernetes.io~projected/kube-api-access-xs8x4 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5d42a36-81a2-49ad-9fa0-32d42e87d75d/volumes/kubernetes.io~projected/kube-api-access-tsrlt:{mountpoint:/var/lib/kubelet/pods/c5d42a36-81a2-49ad-9fa0-32d42e87d75d/volumes/kubernetes.io~projected/kube-api-access-tsrlt major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca9b24b8-5841-4f54-aa38-518bfbb701c6/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ca9b24b8-5841-4f54-aa38-518bfbb701c6/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca9b24b8-5841-4f54-aa38-518bfbb701c6/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ca9b24b8-5841-4f54-aa38-518bfbb701c6/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca9b24b8-5841-4f54-aa38-518bfbb701c6/volumes/kubernetes.io~projected/kube-api-access-vgfgs:{mountpoint:/var/lib/kubelet/pods/ca9b24b8-5841-4f54-aa38-518bfbb701c6/volumes/kubernetes.io~projected/kube-api-access-vgfgs major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb0c5682-acc6-4029-831d-b6aa9027db25/volumes/kubernetes.io~projected/kube-api-access-cbq75:{mountpoint:/var/lib/kubelet/pods/cb0c5682-acc6-4029-831d-b6aa9027db25/volumes/kubernetes.io~projected/kube-api-access-cbq75 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb0c5682-acc6-4029-831d-b6aa9027db25/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/cb0c5682-acc6-4029-831d-b6aa9027db25/volumes/kubernetes.io~secret/default-certificate major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb0c5682-acc6-4029-831d-b6aa9027db25/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/cb0c5682-acc6-4029-831d-b6aa9027db25/volumes/kubernetes.io~secret/stats-auth major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd6d2859-2141-401c-825d-b6f12bb72644/volumes/kubernetes.io~projected/kube-api-access-tm4tl:{mountpoint:/var/lib/kubelet/pods/cd6d2859-2141-401c-825d-b6f12bb72644/volumes/kubernetes.io~projected/kube-api-access-tm4tl major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d28e48a8-f70c-4949-b407-4c0e632f6a35/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d28e48a8-f70c-4949-b407-4c0e632f6a35/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3d2494a-0ef0-4da6-b6dd-225f750bb909/volumes/kubernetes.io~projected/kube-api-access-mkwzt:{mountpoint:/var/lib/kubelet/pods/d3d2494a-0ef0-4da6-b6dd-225f750bb909/volumes/kubernetes.io~projected/kube-api-access-mkwzt major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6516df6-4df4-4213-890a-0b25e274e606/volumes/kubernetes.io~projected/kube-api-access-d49cl:{mountpoint:/var/lib/kubelet/pods/d6516df6-4df4-4213-890a-0b25e274e606/volumes/kubernetes.io~projected/kube-api-access-d49cl major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8/volumes/kubernetes.io~projected/kube-api-access-dc5jg:{mountpoint:/var/lib/kubelet/pods/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8/volumes/kubernetes.io~projected/kube-api-access-dc5jg major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfa424b6-593b-44ac-8e28-c9ba0e3207e8/volumes/kubernetes.io~projected/kube-api-access-c947k:{mountpoint:/var/lib/kubelet/pods/dfa424b6-593b-44ac-8e28-c9ba0e3207e8/volumes/kubernetes.io~projected/kube-api-access-c947k major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9/volumes/kubernetes.io~projected/kube-api-access-vtmxt:{mountpoint:/var/lib/kubelet/pods/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9/volumes/kubernetes.io~projected/kube-api-access-vtmxt major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f697930c-fefc-42f2-835e-7b41cf115aaf/volumes/kubernetes.io~projected/kube-api-access-smpsr:{mountpoint:/var/lib/kubelet/pods/f697930c-fefc-42f2-835e-7b41cf115aaf/volumes/kubernetes.io~projected/kube-api-access-smpsr major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f697930c-fefc-42f2-835e-7b41cf115aaf/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/f697930c-fefc-42f2-835e-7b41cf115aaf/volumes/kubernetes.io~secret/cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/volumes/kubernetes.io~projected/kube-api-access-4kfst:{mountpoint:/var/lib/kubelet/pods/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/volumes/kubernetes.io~projected/kube-api-access-4kfst major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4aaf2a9db0f906b223ee5f9fd3f4cb63ff635d5bef87e1924abc7b8ef3f6913e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/75e2220ceb9cad6d3ac22fe809badfe6bef47ba90ff5dcebef90c8c1bf301037/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1a1941bd5150e7bef2b48474bd95db4d6abe013e7c76d3312a7890055c680210/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/0ec187493c9e535d4ddc509e5af97829af6d81c1fad21f260129c472995a4e9b/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/5562bfafa12497b64a88c6f46058043c6229bca3c185c10a6efc14da6953509b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/01d336c992aa4c60e9a64317b6b4c99eb7aa08b5aad1f56545a23ab9f270342b/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/605115a5c6ad51771bacc233854614bb713cb2888dfdb63b41632732aaa28b3a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-128:{mountpoint:/var/lib/containers/storage/overlay/9b3106ec182c118e807b2a3c14d0e4afd19b79883a07a82246c9cb19fbc5bc65/merged major:0 minor:128 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/6bbd0b29eff3e43601d6998ba301d0285a6c62b2fc5ac4b705dbe8230abd8e45/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/1871cdcd53b058c72281dbc024e0a10425aae2f0448a9c7e750b31314fb639dd/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/5040514d70ddbb6b8b236d06cee8a5fcc10bac432b33c7f697c923dae5ea7381/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/308d843f19937eef74782f346791e6d18402ca89b049fa91db7b36fc6500c062/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/fdbd1109caff20fc419ecc237f65225adb2a3c99dcee411760e74cb0f9d0fa52/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/0fbc90e6c2b4823a10c335d59f43fd8d0dbac7c3714223fd188e6e31ed22a2b7/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/4caa64c33fdab659b82646c631ea5ea4b13b09fc8bbc49bd7b64abf13bd18ae2/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/26fe48c6f7ff6d16544483e462a9e3a7f1837f8dda23650720bb1501b101711b/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/5d0b68614743681b8a23b18cad936b0596d7f11f288704d62301ff51bfa0c57d/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/82c2641035c8673c075d225e61f561743ee93e78544618f7e8cd8ff3d6bf8c22/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/86cbd95f2fe2f95f9d3000be42eb12363abf82f54eaa9e2b0e4855bc13eba135/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/dd3effda63497b747d1923c215ac9d6cfdfcba8382802626af3693c36da2d232/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/838432399a46ee488957d6d278c86302d960c93b25929b91c22a33e2645577ab/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/9bd531a2018c6c005c0ae72a556e6ceea623be0aa2b7c53e4980f98d5ee58cd0/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/25319cde5dddb8b20a78f119277a23de00428719afcbc10725d368fee157afae/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/f5527f2b066806fde53fc8737e50b9edb033103f1a313587c6390578a9d550a7/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/c2c78c1ee9238620ee6ab714e4bfa4cff434b69f451c39035f40eabda075b9e7/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/a0da28b04e92812bfbf2e2e601ea83d5de4ff90aeb2473a77b8fd1eaff525974/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/14a25c721c1dcea26fdf06eadf715e6a8e4f48b1d4f5fd9d5163b8080c3f76ca/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/66a9913b3ca98a2d61d4310f89e625cf557fd893cca106c79691f3f103e6540a/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/c24e1bb25f6141832eb784a929fe50603f4351ab20a5875ff71f62d399b488db/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/a5128cc6ee90495c9e9dca1ad878dd9ce48e8bd55ab01ebb5c08ddfcc43acc1b/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/b66d6cc14ef18c73705a10eb6b0df7ede83868b8e1d7ecd5a7bcc0759b4b070f/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/055bf123e4fb60f68f8adf7c4895d1e34e3ab47c078eaa7d5c5d7f38383ceaef/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/ba8435ac9757983aa1b74c2bab664253eb054db3e8c3274d1062de09d0faf52c/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/d31f724a2753343219c20ff1fe0e359d8ba3ecf0a42cceda03e01e8b9bf7e3fc/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/a112abd6760d1651f53f88d089694fe4c13d93f38145449564415003e4c81283/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/4f52c9763e58eef22a793a38e52d9dbd43fb44b82908ba9adabaa89434313f57/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/a29efb3362c7a1ce9934e215f3f24d5ec8b5eeae726d9750a401525ccf4c64f0/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/630e95b2cea7ee47ec0ee1f3064264bb415738d0f7fe496a9b5ca90768ccd435/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/38e6576a2a465ba106fdf8d8ebbeb3fe5840d16f484a9180f38bd3a79d7d8aba/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/646b9f5d8c8efd999ff6932fd23f9fdff4e40233000e5ec59067caf9dbf4a358/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/47902ea0f53ebda276e70597044277f07520d5fcd85765ecb56a4fffcf27fbf7/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/e0698bab542d03c9269a64e02b469c3e2f025ff4d93a1c17fd1998d22daa9c99/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/a646f0337e37b03a483ee870313664c0e6c02e49db666980626849fc5f377ef4/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/aab8bac8a8c1015f01d89c5caa94d7502c1e0c9582181b1095dcaff84c028644/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/315c5de8ce400ae15ecaf5e19f0af822414a4559a315d7c13e20f396be2c6330/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/fb675e8db340cf27baeb1870c93579ad0d0419d4ac708de0b8d777da3efe851c/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/84a41ea0e7aaec82dee390786c3de201de8648d2fc596888d0f4fbb50f7862c2/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/0a912ff334014d7433f8bc14a2d5b9152bb074e39e1140e9985b794e3321e2ce/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/a5908ac5fe3b65fefa04892a8d0a27798d558abdcf5e628e6bc30fd0b7aedf78/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/91600b53fd0734f25a86dc958d3c88b397c75a2f42ab1ba000a03855ed377974/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/e2dae46c7913dbf2ee430125f29cc53af057eacd4e9f6aa3ff0dfa41e14de8df/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/9d003a7c2597dbe48adba58bab131c062a9929f64c5c2380f870c8dabf377325/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/157b0c85001655f136f9885fda69cd0983f82b9e98eb3de27154616529170049/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/ff374406b4fa60bc6e4cc754310fb2ee724c9b60461060fb9e62933b21603de3/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/bd00ae6a0409373e53a255ca6afe34c70c47a2001aed4d1257ddeb7589d563fd/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/2f7e38a3cdedf6f91f7bbadfa4af335cd0232fba1edbae6f8fcc38438387e1b1/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 02:41:59.445199 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.442281    4960 manager.go:217] Machine: {Timestamp:2026-10-30 02:41:59.441487113 +0000 UTC m=+0.090535525 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100126 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c48d397284ee09c574b7bc6acc833 SystemUUID:ec2c48d3-9728-4ee0-9c57-4b7bc6acc833 BootID:7d9aff31-d605-4577-82f3-83959bba5a40 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d28e48a8-f70c-4949-b407-4c0e632f6a35/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2fb4e9ea19f875c664825b0926b6a616905fc4f12ef6853c726a5394985bf8e/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a3ebba8-b807-4256-b552-5a18ec8412cb/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:294 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/06715a40-fb77-4b9c-89e1-0aefe7b12a72/volumes/kubernetes.io~projected/kube-api-access-jxg9s DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/180be8420badf19cba5a9a9e962f858c4bc29b011327502f30c53c0a27010815/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca9b24b8-5841-4f54-aa38-518bfbb701c6/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02d8d231e9a19a0f506667e517f3cdf3993ea52b4b74c83b8a88a120b220d04c/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca9b24b8-5841-4f54-aa38-518bfbb701c6/volumes/kubernetes.io~projected/kube-api-access-vgfgs DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7a3b1ae985324333bafc677eb2ce527fe499935b2c7ab6fabc34cc6f7b16e37/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7f106bdfbea9cabe563d38be39369a6d72fb0bf702cfc49e8d00ffd79438c69/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75211257-e617-40de-bf22-878d736448d1/volumes/kubernetes.io~projected/kube-api-access-lpt6g DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/volumes/kubernetes.io~projected/kube-api-access-4kfst DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/38e84dd1-8ee0-4a74-8802-3e3200c96642/volumes/kubernetes.io~projected/kube-api-access-cshhd DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba00258135664cc71290793d776279e6bbefbfbfd7c99a5bcdc880a155ba2c7a/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3c022b3d-db3f-45a7-b6b7-5404c6f170c8/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7/volumes/kubernetes.io~projected/kube-api-access-lpf67 DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b93ba591-108f-41bd-afed-012dcd2f8690/volumes/kubernetes.io~projected/kube-api-access-hmg6l DeviceMajor:0 DeviceMinor:220 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cb0c5682-acc6-4029-831d-b6aa9027db25/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f697930c-fefc-42f2-835e-7b41cf115aaf/volumes/kubernetes.io~projected/kube-api-access-smpsr DeviceMajor:0 DeviceMinor:221 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/753ec90d7ab7e4722c76de269ba3c7110ee43e363062e9fd3e563965414d2bda/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ca9b24b8-5841-4f54-aa38-518bfbb701c6/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d6516df6-4df4-4213-890a-0b25e274e606/volumes/kubernetes.io~projected/kube-api-access-d49cl DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6fc49c2f7adc853e0db6bde219654ea6290fed8a8b397ef2ac48964c122a386/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dfa424b6-593b-44ac-8e28-c9ba0e3207e8/volumes/kubernetes.io~projected/kube-api-access-c947k DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1cf080d-9f7d-4614-b446-76d174d9c3f7/volumes/kubernetes.io~projected/kube-api-access-xs8x4 DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b93ba591-108f-41bd-afed-012dcd2f8690/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cb0c5682-acc6-4029-831d-b6aa9027db25/volumes/kubernetes.io~projected/kube-api-access-cbq75 DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f83389d86901aa144c3416cb26dd8e5d69a416640db5e895259a44226353e59/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/67b82271ccb3ea645450dcfdbe744c68725094663336981be1d336bf4de228c8/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91181e4e17e08fed6169aeca64410d61a4ff599dcd8c981d467db052e2d24cdb/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd6d2859-2141-401c-825d-b6f12bb72644/volumes/kubernetes.io~projected/kube-api-access-tm4tl DeviceMajor:0 DeviceMinor:329 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8/volumes/kubernetes.io~projected/kube-api-access-dc5jg DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/410a6e7ef4e47afad48737d7e85659ac5aa54e1fb2e993f98cfd49d5410a53d8/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9/volumes/kubernetes.io~projected/kube-api-access-vtmxt DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f697930c-fefc-42f2-835e-7b41cf115aaf/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3c022b3d-db3f-45a7-b6b7-5404c6f170c8/volumes/kubernetes.io~projected/kube-api-access-sl9cq DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/56d5467708c3d3bc9584847c948764546d4391ddc890c1cb4989e7cb17aa05e5/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66b0866bd0781a66cf486595a939b5ec710e86dc87d6021b5f57cbff2c2f2a7a/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/639d650dbcaa5a892eaaf854b9eb42ddb3692ca7e6d6e99b7ce5b9e7e9954633/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cb0c5682-acc6-4029-831d-b6aa9027db25/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/88d453b9-7e71-4561-8e1b-4b4074651675/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32a22c148d94150ee483058da90f16b3b35fbf12fc9824acf14aa476872056c4/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1337da2-13ee-4b79-9a84-37ef217ae10f/volumes/kubernetes.io~projected/kube-api-access-6j6pt DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e31c17097aecc157404375a3654588a06aae2b3b39f29448f0da4bae02470c79/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f28ccef2af82f377429f80311d72cbb000f56833b5b5376c0eda08d7be2fc126/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5668c9636dbb9aff7816e1059ac0e651e49fb53431a60899b0f30e2240b0c1bb/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/60924e48-a5e3-4f2b-ab35-48479c3b730b/volumes/kubernetes.io~projected/kube-api-access-lqbmt DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb/volumes/kubernetes.io~projected/kube-api-access-dxvfq DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ecdf5dc72cb424d2f4b3cd5ed9fd3beea31f652ce5a66a5f2954ba9dfded6365/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-128 DeviceMajor:0 DeviceMinor:128 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b93ba591-108f-41bd-afed-012dcd2f8690/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b93ba591-108f-41bd-afed-012dcd2f8690/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/88d453b9-7e71-4561-8e1b-4b4074651675/volumes/kubernetes.io~projected/kube-api-access-svt6f DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3d2494a-0ef0-4da6-b6dd-225f750bb909/volumes/kubernetes.io~projected/kube-api-access-mkwzt DeviceMajor:0 DeviceMinor:288 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6218f687ee9a5ad9e74be00e02ab9145085a27d8dffebf2424985e9f4f355b79/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/49a36e47-9ab6-49f4-bde8-85b78ba7c208/volumes/kubernetes.io~projected/kube-api-access-6sv8m DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da/volumes/kubernetes.io~projected/kube-api-access-bs9h4 DeviceMajor:0 DeviceMinor:312 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/895d8ed4-6421-4018-a630-fb995aadf0bc/volumes/kubernetes.io~projected/kube-api-access-6kh57 DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3c022b3d-db3f-45a7-b6b7-5404c6f170c8/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/c5d42a36-81a2-49ad-9fa0-32d42e87d75d/volumes/kubernetes.io~projected/kube-api-access-tsrlt DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0d2d9d6b-2705-49d3-b3d2-079d6d85148f/volumes/kubernetes.io~projected/kube-api-access-tpm79 DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56c4a0de-1d41-4b83-9bac-ddee3f43879b/volumes/kubernetes.io~projected/kube-api-access-jnqw2 DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/438045a0b01f1568bab7043e3ba8e47cef5fb051e72d0adb845357919c6dbd15/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d98adb609a438f29cc18c1911ff35f2e0c45c42c5078807bb7927506095c6c3/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e8de64a-fd25-4aa6-9160-b25e09d116aa/volumes/kubernetes.io~projected/kube-api-access-wl6q6 DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:02d8d231e9a19a0 MacAddress:02:9f:c6:60:22:43 Speed:10000 Mtu:8901} {Name:180be8420badf19 MacAddress:06:90:bd:e6:29:6c Speed:10000 Mtu:8901} {Name:410a6e7ef4e47af MacAddress:ca:35:d4:bd:da:e8 Speed:10000 Mtu:8901} {Name:5668c9636dbb9af MacAddress:22:27:19:48:0a:73 Speed:10000 Mtu:8901} {Name:56d5467708c3d3b MacAddress:3e:b4:5a:99:d5:fc Speed:10000 Mtu:8901} {Name:66b0866bd0781a6 MacAddress:aa:da:dc:00:bb:08 Speed:10000 Mtu:8901} {Name:753ec90d7ab7e47 MacAddress:66:18:0d:b6:6d:b1 Speed:10000 Mtu:8901} {Name:91181e4e17e08fe MacAddress:22:ed:1a:ea:17:43 Speed:10000 Mtu:8901} {Name:ba00258135664cc MacAddress:9e:25:ca:cc:02:f4 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d1:85:a6:fd Speed:0 Mtu:9001} {Name:br-int MacAddress:ca:2e:59:8a:4a:18 Speed:0 Mtu:8901} {Name:c7f106bdfbea9ca MacAddress:aa:07:84:cc:85:09 Speed:10000 Mtu:8901} {Name:e31c17097aecc15 MacAddress:92:1e:19:32:0a:43 Speed:10000 Mtu:8901} {Name:e6fc49c2f7adc85 MacAddress:76:48:2b:b9:25:01 Speed:10000 Mtu:8901} {Name:e7a3b1ae9853243 MacAddress:02:66:ef:9a:34:19 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d1:85:a6:fd Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:6e:c3:cb:c5:76:96 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ae:a6:00:54:41:1a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:41:59.445199 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.442608    4960 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:41:59.445199 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.442663    4960 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:41:59.445199 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.442990    4960 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:41:59.445199 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.443041    4960 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-241.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:41:59.445199 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.443240    4960 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:41:59.445199 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.443252    4960 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:41:59.445199 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.443268    4960 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:41:59.445199 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.443310    4960 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:41:59.445199 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.443413    4960 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:41:59.445199 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.443620    4960 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:41:59.445199 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.443758    4960 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:41:59.445199 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.443804    4960 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:41:59.445199 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.443823    4960 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:41:59.445199 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.443836    4960 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:41:59.445199 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.443849    4960 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:41:59.445199 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.443859    4960 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:41:59.445199 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.444740    4960 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:41:59.445199 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.444754    4960 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:41:59.446344 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.445245    4960 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:41:59.447775 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.447756    4960 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:41:59.447775 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.447772    4960 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:41:59.447921 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.447908    4960 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 02:41:59.448135 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.448119    4960 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:41:59.448216 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.448159    4960 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:41:59.448321 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.448307    4960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:41:59.448375 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.448335    4960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:41:59.448375 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.448346    4960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:41:59.448375 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.448356    4960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:41:59.448375 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.448365    4960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:41:59.448375 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.448375    4960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:41:59.448602 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.448384    4960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:41:59.448602 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.448393    4960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:41:59.448602 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.448411    4960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:41:59.448602 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.448420    4960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:41:59.448602 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.448439    4960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:41:59.448602 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.448454    4960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:41:59.448602 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.448498    4960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:41:59.448602 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.448508    4960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:41:59.452025 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.452008    4960 server.go:1263] "Started kubelet"
Oct 30 02:41:59.452428 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.452168    4960 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:41:59.452564 ip-10-0-128-241 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:41:59.452781 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.452725    4960 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:41:59.452934 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.452812    4960 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:41:59.453172 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.453136    4960 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:41:59.453981 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.453965    4960 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:41:59.457951 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.457937    4960 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:41:59.458031 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.458013    4960 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.458172    4960 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.458215    4960 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.458332    4960 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.458326    4960 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:36:21 +0000 UTC" deadline="2028-04-22 21:10:57.186188728 +0000 UTC"
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.458356    4960 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12978h28m57.72783925s"
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:41:59.458593    4960 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-241.ec2.internal\" not found"
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.458722    4960 factory.go:55] Registering systemd factory
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.458740    4960 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.461666    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c022b3d-db3f-45a7-b6b7-5404c6f170c8" volumeName="kubernetes.io/configmap/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-ovnkube-config" seLinuxMountContext=""
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.461716    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c022b3d-db3f-45a7-b6b7-5404c6f170c8" volumeName="kubernetes.io/configmap/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.461736    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49a36e47-9ab6-49f4-bde8-85b78ba7c208" volumeName="kubernetes.io/configmap/49a36e47-9ab6-49f4-bde8-85b78ba7c208-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.461755    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e8de64a-fd25-4aa6-9160-b25e09d116aa" volumeName="kubernetes.io/projected/7e8de64a-fd25-4aa6-9160-b25e09d116aa-kube-api-access-wl6q6" seLinuxMountContext=""
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.461766    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b93ba591-108f-41bd-afed-012dcd2f8690" volumeName="kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-bound-sa-token" seLinuxMountContext=""
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.461782    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b93ba591-108f-41bd-afed-012dcd2f8690" volumeName="kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-installation-pull-secrets" seLinuxMountContext=""
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.461806    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5d42a36-81a2-49ad-9fa0-32d42e87d75d" volumeName="kubernetes.io/empty-dir/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-tmp-dir" seLinuxMountContext=""
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.461822    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd6d2859-2141-401c-825d-b6f12bb72644" volumeName="kubernetes.io/projected/cd6d2859-2141-401c-825d-b6f12bb72644-kube-api-access-tm4tl" seLinuxMountContext=""
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.461833    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88d453b9-7e71-4561-8e1b-4b4074651675" volumeName="kubernetes.io/configmap/88d453b9-7e71-4561-8e1b-4b4074651675-config" seLinuxMountContext=""
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.461861    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1cf080d-9f7d-4614-b446-76d174d9c3f7" volumeName="kubernetes.io/projected/c1cf080d-9f7d-4614-b446-76d174d9c3f7-kube-api-access-xs8x4" seLinuxMountContext=""
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.461889    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9" volumeName="kubernetes.io/projected/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-kube-api-access-vtmxt" seLinuxMountContext=""
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.461903    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f697930c-fefc-42f2-835e-7b41cf115aaf" volumeName="kubernetes.io/secret/f697930c-fefc-42f2-835e-7b41cf115aaf-cert" seLinuxMountContext=""
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.461922    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b93ba591-108f-41bd-afed-012dcd2f8690" volumeName="kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-trusted-ca" seLinuxMountContext=""
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.461941    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="895d8ed4-6421-4018-a630-fb995aadf0bc" volumeName="kubernetes.io/projected/895d8ed4-6421-4018-a630-fb995aadf0bc-kube-api-access-6kh57" seLinuxMountContext=""
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.461980    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d2d9d6b-2705-49d3-b3d2-079d6d85148f" volumeName="kubernetes.io/projected/0d2d9d6b-2705-49d3-b3d2-079d6d85148f-kube-api-access-tpm79" seLinuxMountContext=""
Oct 30 02:41:59.461992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462003    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56c4a0de-1d41-4b83-9bac-ddee3f43879b" volumeName="kubernetes.io/configmap/56c4a0de-1d41-4b83-9bac-ddee3f43879b-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462029    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" volumeName="kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-trusted-ca" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462043    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5d42a36-81a2-49ad-9fa0-32d42e87d75d" volumeName="kubernetes.io/configmap/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-config-volume" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462051    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5d42a36-81a2-49ad-9fa0-32d42e87d75d" volumeName="kubernetes.io/projected/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-kube-api-access-tsrlt" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462064    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f697930c-fefc-42f2-835e-7b41cf115aaf" volumeName="kubernetes.io/projected/f697930c-fefc-42f2-835e-7b41cf115aaf-kube-api-access-smpsr" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462071    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88d453b9-7e71-4561-8e1b-4b4074651675" volumeName="kubernetes.io/projected/88d453b9-7e71-4561-8e1b-4b4074651675-kube-api-access-svt6f" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462092    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="895d8ed4-6421-4018-a630-fb995aadf0bc" volumeName="kubernetes.io/configmap/895d8ed4-6421-4018-a630-fb995aadf0bc-iptables-alerter-script" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462108    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9" volumeName="kubernetes.io/configmap/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-service-ca-bundle" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462116    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c022b3d-db3f-45a7-b6b7-5404c6f170c8" volumeName="kubernetes.io/projected/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-kube-api-access-sl9cq" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462129    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56c4a0de-1d41-4b83-9bac-ddee3f43879b" volumeName="kubernetes.io/configmap/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-daemon-config" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462139    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da" volumeName="kubernetes.io/projected/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-api-access-bs9h4" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462155    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88d453b9-7e71-4561-8e1b-4b4074651675" volumeName="kubernetes.io/secret/88d453b9-7e71-4561-8e1b-4b4074651675-serving-cert" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462166    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" volumeName="kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-installation-pull-secrets" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462197    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb0c5682-acc6-4029-831d-b6aa9027db25" volumeName="kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-default-certificate" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462205    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6516df6-4df4-4213-890a-0b25e274e606" volumeName="kubernetes.io/projected/d6516df6-4df4-4213-890a-0b25e274e606-kube-api-access-d49cl" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462214    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8" volumeName="kubernetes.io/projected/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8-kube-api-access-dc5jg" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462225    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38e84dd1-8ee0-4a74-8802-3e3200c96642" volumeName="kubernetes.io/empty-dir/38e84dd1-8ee0-4a74-8802-3e3200c96642-tmp-dir" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462238    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2796ed5-6fd6-4231-9847-1bee8c9f9dd7" volumeName="kubernetes.io/configmap/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7-serviceca" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462267    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b93ba591-108f-41bd-afed-012dcd2f8690" volumeName="kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-kube-api-access-hmg6l" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462275    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" volumeName="kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462284    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb0c5682-acc6-4029-831d-b6aa9027db25" volumeName="kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-stats-auth" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462291    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9" volumeName="kubernetes.io/empty-dir/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-snapshots" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462302    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e" volumeName="kubernetes.io/configmap/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-trusted-ca" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462315    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38e84dd1-8ee0-4a74-8802-3e3200c96642" volumeName="kubernetes.io/projected/38e84dd1-8ee0-4a74-8802-3e3200c96642-kube-api-access-cshhd" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462330    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49a36e47-9ab6-49f4-bde8-85b78ba7c208" volumeName="kubernetes.io/configmap/49a36e47-9ab6-49f4-bde8-85b78ba7c208-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462343    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56c4a0de-1d41-4b83-9bac-ddee3f43879b" volumeName="kubernetes.io/projected/56c4a0de-1d41-4b83-9bac-ddee3f43879b-kube-api-access-jnqw2" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462350    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e" volumeName="kubernetes.io/configmap/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-config" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462359    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e" volumeName="kubernetes.io/projected/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-kube-api-access-4kfst" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462368    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60924e48-a5e3-4f2b-ab35-48479c3b730b" volumeName="kubernetes.io/projected/60924e48-a5e3-4f2b-ab35-48479c3b730b-kube-api-access-lqbmt" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462375    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a3ebba8-b807-4256-b552-5a18ec8412cb" volumeName="kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462388    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca9b24b8-5841-4f54-aa38-518bfbb701c6" volumeName="kubernetes.io/empty-dir/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-tuned" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462395    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb0c5682-acc6-4029-831d-b6aa9027db25" volumeName="kubernetes.io/projected/cb0c5682-acc6-4029-831d-b6aa9027db25-kube-api-access-cbq75" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462404    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8" volumeName="kubernetes.io/secret/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8-serving-cert" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462435    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06715a40-fb77-4b9c-89e1-0aefe7b12a72" volumeName="kubernetes.io/configmap/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-config" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462448    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2796ed5-6fd6-4231-9847-1bee8c9f9dd7" volumeName="kubernetes.io/projected/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7-kube-api-access-lpf67" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462459    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b93ba591-108f-41bd-afed-012dcd2f8690" volumeName="kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-registry-certificates" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462469    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d28e48a8-f70c-4949-b407-4c0e632f6a35" volumeName="kubernetes.io/configmap/d28e48a8-f70c-4949-b407-4c0e632f6a35-konnectivity-ca" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462478    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9" volumeName="kubernetes.io/secret/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-serving-cert" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462485    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c022b3d-db3f-45a7-b6b7-5404c6f170c8" volumeName="kubernetes.io/secret/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462496    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" volumeName="kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-kube-api-access-dxvfq" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462508    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3d2494a-0ef0-4da6-b6dd-225f750bb909" volumeName="kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462534    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8" volumeName="kubernetes.io/configmap/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8-config" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462546    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9" volumeName="kubernetes.io/empty-dir/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-tmp" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462553    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e" volumeName="kubernetes.io/secret/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-serving-cert" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462566    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06715a40-fb77-4b9c-89e1-0aefe7b12a72" volumeName="kubernetes.io/projected/06715a40-fb77-4b9c-89e1-0aefe7b12a72-kube-api-access-jxg9s" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462576    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49a36e47-9ab6-49f4-bde8-85b78ba7c208" volumeName="kubernetes.io/projected/49a36e47-9ab6-49f4-bde8-85b78ba7c208-kube-api-access-6sv8m" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462591    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b93ba591-108f-41bd-afed-012dcd2f8690" volumeName="kubernetes.io/empty-dir/b93ba591-108f-41bd-afed-012dcd2f8690-ca-trust-extracted" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462601    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" volumeName="kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-bound-sa-token" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462611    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca9b24b8-5841-4f54-aa38-518bfbb701c6" volumeName="kubernetes.io/projected/ca9b24b8-5841-4f54-aa38-518bfbb701c6-kube-api-access-vgfgs" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462631    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da" volumeName="kubernetes.io/empty-dir/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-data-volume" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462645    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75211257-e617-40de-bf22-878d736448d1" volumeName="kubernetes.io/projected/75211257-e617-40de-bf22-878d736448d1-kube-api-access-lpt6g" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462658    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca9b24b8-5841-4f54-aa38-518bfbb701c6" volumeName="kubernetes.io/empty-dir/ca9b24b8-5841-4f54-aa38-518bfbb701c6-tmp" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462670    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfa424b6-593b-44ac-8e28-c9ba0e3207e8" volumeName="kubernetes.io/projected/dfa424b6-593b-44ac-8e28-c9ba0e3207e8-kube-api-access-c947k" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462677    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9" volumeName="kubernetes.io/configmap/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462687    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b93ba591-108f-41bd-afed-012dcd2f8690" volumeName="kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462696    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6516df6-4df4-4213-890a-0b25e274e606" volumeName="kubernetes.io/configmap/d6516df6-4df4-4213-890a-0b25e274e606-telemetry-config" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462707    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49a36e47-9ab6-49f4-bde8-85b78ba7c208" volumeName="kubernetes.io/configmap/49a36e47-9ab6-49f4-bde8-85b78ba7c208-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462719    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70f081e1-7202-4b86-930d-8c3f72a3bb7e" volumeName="kubernetes.io/configmap/70f081e1-7202-4b86-930d-8c3f72a3bb7e-nginx-conf" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462729    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" volumeName="kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-certificates" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462747    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d28e48a8-f70c-4949-b407-4c0e632f6a35" volumeName="kubernetes.io/secret/d28e48a8-f70c-4949-b407-4c0e632f6a35-agent-certs" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462761    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c022b3d-db3f-45a7-b6b7-5404c6f170c8" volumeName="kubernetes.io/configmap/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-env-overrides" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462768    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1337da2-13ee-4b79-9a84-37ef217ae10f" volumeName="kubernetes.io/projected/b1337da2-13ee-4b79-9a84-37ef217ae10f-kube-api-access-6j6pt" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462777    4960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" volumeName="kubernetes.io/empty-dir/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-ca-trust-extracted" seLinuxMountContext=""
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462785    4960 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:41:59.463143 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.462789    4960 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:41:59.466108 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.464172    4960 factory.go:153] Registering CRI-O factory
Oct 30 02:41:59.466108 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.464217    4960 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:41:59.466108 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.464295    4960 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:41:59.466108 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.464313    4960 factory.go:103] Registering Raw factory
Oct 30 02:41:59.466108 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.464327    4960 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:41:59.466108 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.464773    4960 manager.go:319] Starting recovery of all containers
Oct 30 02:41:59.484096 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.484069    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/1.log"
Oct 30 02:41:59.484989 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.484972    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/1.log"
Oct 30 02:41:59.494278 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.494251    4960 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:41:59.495335 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.495302    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 02:41:59.496157 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.496134    4960 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:41:59.496157 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.496160    4960 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:41:59.496308 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.496204    4960 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:41:59.496308 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:41:59.496247    4960 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 02:41:59.498141 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.498112    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 02:41:59.511654 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.511635    4960 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 02:41:59.513974 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.513949    4960 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:59.514257 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.514235    4960 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:59.515339 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.515311    4960 generic.go:356] "Generic (PLEG): container finished" podID="49a36e47-9ab6-49f4-bde8-85b78ba7c208" containerID="8b089dca29ad4871f6d5b1917646d47654ef68c40be7b73fca8c6bf80d173f18" exitCode=0
Oct 30 02:41:59.515339 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.515338    4960 generic.go:356] "Generic (PLEG): container finished" podID="49a36e47-9ab6-49f4-bde8-85b78ba7c208" containerID="55793a574377f734b917dad5f8cf2a699314270fe3c4a8d8c69224a8da0577bd" exitCode=0
Oct 30 02:41:59.515453 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.515348    4960 generic.go:356] "Generic (PLEG): container finished" podID="49a36e47-9ab6-49f4-bde8-85b78ba7c208" containerID="c6788018b54c1b9f0330524652016330ef1b1881cec4a1d3f10b1c9307eae839" exitCode=0
Oct 30 02:41:59.515453 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.515357    4960 generic.go:356] "Generic (PLEG): container finished" podID="49a36e47-9ab6-49f4-bde8-85b78ba7c208" containerID="d9fe334ef5a800fb4e897389e8b4d44ca9c131908f1b0bfa021c65491f778ae6" exitCode=0
Oct 30 02:41:59.515453 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.515369    4960 generic.go:356] "Generic (PLEG): container finished" podID="49a36e47-9ab6-49f4-bde8-85b78ba7c208" containerID="74b4bbd24ffe5a1ac85ea66f35814e899d37d84743566d1f40181fe3907cf8c7" exitCode=0
Oct 30 02:41:59.515453 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.515381    4960 generic.go:356] "Generic (PLEG): container finished" podID="49a36e47-9ab6-49f4-bde8-85b78ba7c208" containerID="7b9f41d0ceb1ebf685a9930181567c76fb529f39b89765c7478c8837c69c3242" exitCode=0
Oct 30 02:41:59.523848 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.523829    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 02:41:59.524221 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.524200    4960 generic.go:356] "Generic (PLEG): container finished" podID="3c022b3d-db3f-45a7-b6b7-5404c6f170c8" containerID="a9f99c54021d76c2a97d213bdb0ad3f0cb93d53eaf6b1fb1fee25f25e8e0556a" exitCode=1
Oct 30 02:41:59.527762 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.527744    4960 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:59.527980 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.527964    4960 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:59.532777 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.532729    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/1.log"
Oct 30 02:41:59.532777 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.532765    4960 generic.go:356] "Generic (PLEG): container finished" podID="fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e" containerID="d49244d3852bbacd675ebd114cf975faac54dc9d1ec0822431ad05f17f237c73" exitCode=255
Oct 30 02:41:59.540034 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.540004    4960 generic.go:356] "Generic (PLEG): container finished" podID="0b7cbf3e76b96e5e2b4f68dfa17fdc26" containerID="2896b4b72c34dd2d3a9ecc96dec656d0c7168fa718181d889718debb257591a1" exitCode=0
Oct 30 02:41:59.558979 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.558964    4960 manager.go:324] Recovery completed
Oct 30 02:41:59.579599 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.579582    4960 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:41:59.579599 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.579596    4960 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:41:59.579695 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.579615    4960 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:41:59.579747 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.579720    4960 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 02:41:59.579747 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.579731    4960 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 02:41:59.579747 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.579747    4960 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 02:41:59.579830 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.579753    4960 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 02:41:59.579830 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.579758    4960 policy_none.go:50] "Start"
Oct 30 02:41:59.579830 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.579766    4960 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:41:59.579830 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.579774    4960 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:41:59.579933 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.579886    4960 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 02:41:59.579933 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.579893    4960 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 02:41:59.579933 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.579898    4960 policy_none.go:44] "Start"
Oct 30 02:41:59.584558 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.584536    4960 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:41:59.587123 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.587109    4960 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:41:59.587693 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:41:59.587133    4960 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:41:59.587693 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.587693    4960 server.go:87] "Starting device plugin registration server"
Oct 30 02:41:59.587943 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.587927    4960 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:41:59.588009 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.587943    4960 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:41:59.588059 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.588039    4960 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:41:59.588196 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.588145    4960 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:41:59.588263 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.588208    4960 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:41:59.588263 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.588217    4960 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:41:59.588378 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.588283    4960 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:41:59.588197505Z","UUID":"6358f73e-6525-4e68-b5c3-e4c8c772a3d4","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:41:59.589481 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:41:59.589459    4960 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:41:59.589610 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.589592    4960 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:41:59.589703 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.589618    4960 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:41:59.596353 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.596323    4960 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal"]
Oct 30 02:41:59.597527 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.597499    4960 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:59.597932 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.597879    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal" event={"ID":"2ea871c29059b568921cda32c2dd3ba5","Type":"ContainerStarted","Data":"7c92bf6543ded960e6603ecfd73d56610e94917dd22fc50dcd19131eb3a2e417"}
Oct 30 02:41:59.598022 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.597956    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal" event={"ID":"2ea871c29059b568921cda32c2dd3ba5","Type":"ContainerStarted","Data":"3f83389d86901aa144c3416cb26dd8e5d69a416640db5e895259a44226353e59"}
Oct 30 02:41:59.598077 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.598061    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal" event={"ID":"0b7cbf3e76b96e5e2b4f68dfa17fdc26","Type":"ContainerStarted","Data":"7515a82683aa7c001d2ba067d4f8d6d8b641fbe12303dbd0a533ad2cbe3f801d"}
Oct 30 02:41:59.598127 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.598088    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal" event={"ID":"0b7cbf3e76b96e5e2b4f68dfa17fdc26","Type":"ContainerDied","Data":"2896b4b72c34dd2d3a9ecc96dec656d0c7168fa718181d889718debb257591a1"}
Oct 30 02:41:59.598127 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.598121    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal" event={"ID":"0b7cbf3e76b96e5e2b4f68dfa17fdc26","Type":"ContainerStarted","Data":"6218f687ee9a5ad9e74be00e02ab9145085a27d8dffebf2424985e9f4f355b79"}
Oct 30 02:41:59.598482 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.598461    4960 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:59.625156 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.625137    4960 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:41:59.625250 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:41:59.625226    4960 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-241.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:59.625901 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.625890    4960 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:41:59.625953 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:41:59.625939    4960 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:59.689022 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.689005    4960 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:41:59.690058 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.690043    4960 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:41:59.690123 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.690067    4960 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:41:59.690123 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.690076    4960 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-241.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:41:59.690123 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.690109    4960 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-241.ec2.internal"
Oct 30 02:41:59.712005 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.711986    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2ea871c29059b568921cda32c2dd3ba5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-241.ec2.internal\" (UID: \"2ea871c29059b568921cda32c2dd3ba5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:59.712064 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.712011    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b7cbf3e76b96e5e2b4f68dfa17fdc26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal\" (UID: \"0b7cbf3e76b96e5e2b4f68dfa17fdc26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:59.712064 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.712028    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b7cbf3e76b96e5e2b4f68dfa17fdc26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal\" (UID: \"0b7cbf3e76b96e5e2b4f68dfa17fdc26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:59.730161 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.730142    4960 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-241.ec2.internal"
Oct 30 02:41:59.730246 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.730205    4960 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-241.ec2.internal"
Oct 30 02:41:59.813122 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.813064    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2ea871c29059b568921cda32c2dd3ba5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-241.ec2.internal\" (UID: \"2ea871c29059b568921cda32c2dd3ba5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:59.813122 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.813098    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b7cbf3e76b96e5e2b4f68dfa17fdc26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal\" (UID: \"0b7cbf3e76b96e5e2b4f68dfa17fdc26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:59.813296 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.813125    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b7cbf3e76b96e5e2b4f68dfa17fdc26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal\" (UID: \"0b7cbf3e76b96e5e2b4f68dfa17fdc26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:59.813296 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.813171    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2ea871c29059b568921cda32c2dd3ba5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-241.ec2.internal\" (UID: \"2ea871c29059b568921cda32c2dd3ba5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:59.813296 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.813198    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b7cbf3e76b96e5e2b4f68dfa17fdc26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal\" (UID: \"0b7cbf3e76b96e5e2b4f68dfa17fdc26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal"
Oct 30 02:41:59.813296 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:41:59.813199    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b7cbf3e76b96e5e2b4f68dfa17fdc26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal\" (UID: \"0b7cbf3e76b96e5e2b4f68dfa17fdc26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal"
Oct 30 02:42:00.444963 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.444937    4960 apiserver.go:52] "Watching apiserver"
Oct 30 02:42:00.518310 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.518287    4960 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:42:00.519059 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.519034    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-241.ec2.internal","openshift-cluster-node-tuning-operator/tuned-vx9zb","openshift-image-registry/image-registry-59bd5b5c6b-lcbtn","kube-system/kas-connection-checker-685895c7c9-2sq5m","kube-system/konnectivity-agent-9xpbx","kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal","openshift-dns/dns-default-7mhtz","openshift-network-diagnostics/network-check-target-wx7bz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t","openshift-insights/insights-operator-7c4986d449-gxb72","openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf","openshift-multus/multus-kbn25","openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s","openshift-ovn-kubernetes/ovnkube-node-fl65k","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg","openshift-dns/node-resolver-6pzrk","openshift-ingress-canary/ingress-canary-fx6bk","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-njw6d","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp","kube-system/kas-connection-checker-685895c7c9-dhf78","openshift-insights/insights-runtime-extractor-dkfnj","openshift-multus/multus-additional-cni-plugins-mjph2","kube-system/kas-connection-checker-685895c7c9-jsbwd","openshift-network-console/networking-console-plugin-86c478bb88-jwzx9","openshift-ingress/router-default-5d4bdb4f7f-89p4x","openshift-multus/network-metrics-daemon-t5747","openshift-kube-storage-version-migrator/migrator-66df5d496b-dgtzw","openshift-image-registry/node-ca-tvbs5","openshift-network-diagnostics/network-check-source-6f78d59f99-7hpk4","openshift-image-registry/image-registry-7c6dbd69f-lrwtw","kube-system/global-pull-secret-syncer-skc8q","openshift-network-operator/iptables-alerter-24lkk","openshift-console-operator/console-operator-7cd56b8dbc-scstg"]
Oct 30 02:42:00.520771 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.520753    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:00.524815 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.524773    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:42:00.524918 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.524817    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:42:00.526722 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.526702    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:00.527085 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.527066    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:42:00.529915 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.529900    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:42:00.532607 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.532592    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:00.534208 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.534173    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:42:00.535500 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.535481    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:00.536408 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.536390    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 02:42:00.537122 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.537105    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 02:42:00.537203 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.537168    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 02:42:00.537291 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.537274    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 02:42:00.537345 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.537298    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 02:42:00.537345 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.537300    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 02:42:00.551436 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.551414    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:42:00.551736 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.551714    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:42:00.551736 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.551729    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 02:42:00.551893 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.551740    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 02:42:00.551893 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.551805    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 02:42:00.551893 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.551825    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 02:42:00.551893 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.551862    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 02:42:00.551893 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.551883    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 02:42:00.551893 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.551885    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:42:00.552223 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.551876    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 02:42:00.552223 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.552070    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 02:42:00.552223 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.552078    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 02:42:00.552223 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.552123    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 02:42:00.552223 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.552121    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:42:00.552223 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.552126    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 02:42:00.552223 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.552146    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 02:42:00.552223 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.552156    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 02:42:00.552223 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.552205    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 02:42:00.552223 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.552207    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 02:42:00.552655 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.552252    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 02:42:00.552655 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.552254    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 02:42:00.552655 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.552394    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 02:42:00.552655 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.552570    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:42:00.552655 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.552637    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 02:42:00.552873 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.552662    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:42:00.552873 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.552813    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 02:42:00.552873 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.552816    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 02:42:00.553022 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.552929    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8xpcs\""
Oct 30 02:42:00.553022 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.552949    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qd44t\""
Oct 30 02:42:00.555473 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.555450    4960 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal"
Oct 30 02:42:00.560171 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.560154    4960 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:42:00.565609 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.565587    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 02:42:00.567123 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567090    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 02:42:00.567242 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567137    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 02:42:00.567242 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567225    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-kxfcp\""
Oct 30 02:42:00.567344 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567280    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-29rqz\""
Oct 30 02:42:00.567407 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567368    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mcz9c\""
Oct 30 02:42:00.567407 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567367    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:42:00.567407 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567387    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:42:00.567551 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567428    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:42:00.567551 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567442    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:42:00.567551 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567462    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:42:00.567551 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567534    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 02:42:00.567551 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567535    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 02:42:00.567780 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567561    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:42:00.567780 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567567    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 02:42:00.567780 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567588    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8z95j\""
Oct 30 02:42:00.567780 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567599    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:42:00.567780 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567624    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 02:42:00.567780 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567632    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:42:00.567780 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567634    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:42:00.567780 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567754    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:42:00.567780 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567645    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 02:42:00.567780 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567657    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:42:00.567780 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567783    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 02:42:00.568138 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567660    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 02:42:00.568138 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567661    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:42:00.568138 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567681    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:42:00.568138 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567694    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:42:00.568138 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567826    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:42:00.568138 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567845    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-czntg\""
Oct 30 02:42:00.568138 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567630    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:42:00.568138 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567862    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 02:42:00.568138 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567755    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 02:42:00.568138 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567710    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:42:00.568138 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567721    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:42:00.568138 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567904    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zm8vf\""
Oct 30 02:42:00.568138 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567912    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 02:42:00.568138 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567925    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 02:42:00.568138 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.567982    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 02:42:00.568138 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.568043    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hjgzr\""
Oct 30 02:42:00.568138 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.568104    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-d89pf\""
Oct 30 02:42:00.573689 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.573671    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:00.579096 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.579079    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:42:00.579155 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.579097    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:42:00.579315 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.579303    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:42:00.579491 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.579476    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 02:42:00.579534 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.579508    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:42:00.579534 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.579520    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 02:42:00.579623 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.579546    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:42:00.579623 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.579596    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:42:00.579716 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.579703    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:42:00.579770 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.579722    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:42:00.579815 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.579774    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 02:42:00.579815 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.579779    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:42:00.579815 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.579777    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vzqmp\""
Oct 30 02:42:00.580125 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.580040    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:42:00.580125 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.580077    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6bd55\""
Oct 30 02:42:00.580125 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.580084    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-stwqd\""
Oct 30 02:42:00.580125 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.580086    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tbftn\""
Oct 30 02:42:00.580125 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.580099    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2wtbz\""
Oct 30 02:42:00.580125 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.580105    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zz8vj\""
Oct 30 02:42:00.580125 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.580110    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-dgvq2\""
Oct 30 02:42:00.580467 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.580141    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j74dx\""
Oct 30 02:42:00.580467 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.580198    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:42:00.580467 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.580345    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:42:00.580467 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.580397    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hbwgl\""
Oct 30 02:42:00.580467 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.580426    4960 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:42:00.580675 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.580471    4960 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-241.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-241.ec2.internal"
Oct 30 02:42:00.580841 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.580826    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 02:42:00.617163 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617141    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/88d453b9-7e71-4561-8e1b-4b4074651675-config\") pod \"service-ca-operator-544d749c8c-vbz2s\" (UID: \"88d453b9-7e71-4561-8e1b-4b4074651675\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"
Oct 30 02:42:00.617248 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617166    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-certificates\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:00.617248 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617206    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-serving-cert\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:42:00.617248 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617225    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-kubelet\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.617338 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617302    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cshhd\" (UniqueName: \"kubernetes.io/projected/38e84dd1-8ee0-4a74-8802-3e3200c96642-kube-api-access-cshhd\") pod \"node-resolver-6pzrk\" (UID: \"38e84dd1-8ee0-4a74-8802-3e3200c96642\") " pod="openshift-dns/node-resolver-6pzrk"
Oct 30 02:42:00.617381 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617340    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.617381 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617370    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:00.617469 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617384    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-certificates\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:00.617469 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617402    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/895d8ed4-6421-4018-a630-fb995aadf0bc-iptables-alerter-script\") pod \"iptables-alerter-24lkk\" (UID: \"895d8ed4-6421-4018-a630-fb995aadf0bc\") " pod="openshift-network-operator/iptables-alerter-24lkk"
Oct 30 02:42:00.617469 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617421    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/88d453b9-7e71-4561-8e1b-4b4074651675-config\") pod \"service-ca-operator-544d749c8c-vbz2s\" (UID: \"88d453b9-7e71-4561-8e1b-4b4074651675\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"
Oct 30 02:42:00.617469 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617455    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-serving-cert\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:42:00.617650 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617470    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-stats-auth\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:00.617650 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617509    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl6q6\" (UniqueName: \"kubernetes.io/projected/7e8de64a-fd25-4aa6-9160-b25e09d116aa-kube-api-access-wl6q6\") pod \"volume-data-source-validator-58c9f5f46d-njw6d\" (UID: \"7e8de64a-fd25-4aa6-9160-b25e09d116aa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-njw6d"
Oct 30 02:42:00.617650 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617537    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sv8m\" (UniqueName: \"kubernetes.io/projected/49a36e47-9ab6-49f4-bde8-85b78ba7c208-kube-api-access-6sv8m\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:42:00.617650 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617566    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:42:00.617650 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617592    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg\" (UID: \"dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"
Oct 30 02:42:00.617650 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617622    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-lib-modules\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.617848 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617664    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs8x4\" (UniqueName: \"kubernetes.io/projected/c1cf080d-9f7d-4614-b446-76d174d9c3f7-kube-api-access-xs8x4\") pod \"kas-connection-checker-685895c7c9-dhf78\" (UID: \"c1cf080d-9f7d-4614-b446-76d174d9c3f7\") " pod="kube-system/kas-connection-checker-685895c7c9-dhf78"
Oct 30 02:42:00.617848 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617717    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-stats-auth\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:00.617848 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617782    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm4tl\" (UniqueName: \"kubernetes.io/projected/cd6d2859-2141-401c-825d-b6f12bb72644-kube-api-access-tm4tl\") pod \"migrator-66df5d496b-dgtzw\" (UID: \"cd6d2859-2141-401c-825d-b6f12bb72644\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dgtzw"
Oct 30 02:42:00.617848 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617805    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg\" (UID: \"dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"
Oct 30 02:42:00.617848 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617837    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/895d8ed4-6421-4018-a630-fb995aadf0bc-iptables-alerter-script\") pod \"iptables-alerter-24lkk\" (UID: \"895d8ed4-6421-4018-a630-fb995aadf0bc\") " pod="openshift-network-operator/iptables-alerter-24lkk"
Oct 30 02:42:00.617992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617869    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a3ebba8-b807-4256-b552-5a18ec8412cb-kubelet-config\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:42:00.617992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617911    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca9b24b8-5841-4f54-aa38-518bfbb701c6-tmp\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.617992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.617948    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgfgs\" (UniqueName: \"kubernetes.io/projected/ca9b24b8-5841-4f54-aa38-518bfbb701c6-kube-api-access-vgfgs\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.618084 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618006    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:42:00.618084 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618020    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-cnibin\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.618084 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618013    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca9b24b8-5841-4f54-aa38-518bfbb701c6-tmp\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.618173 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618108    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-run-k8s-cni-cncf-io\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.618173 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618131    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnqw2\" (UniqueName: \"kubernetes.io/projected/56c4a0de-1d41-4b83-9bac-ddee3f43879b-kube-api-access-jnqw2\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.618173 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618150    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-node-log\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.618293 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618174    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-ovn-node-metrics-cert\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.618293 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618226    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-sys\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.618293 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618274    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d6516df6-4df4-4213-890a-0b25e274e606-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:42:00.618386 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618312    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:42:00.618386 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618361    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:00.618462 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618386    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:00.618462 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618401    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/38e84dd1-8ee0-4a74-8802-3e3200c96642-hosts-file\") pod \"node-resolver-6pzrk\" (UID: \"38e84dd1-8ee0-4a74-8802-3e3200c96642\") " pod="openshift-dns/node-resolver-6pzrk"
Oct 30 02:42:00.618462 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618417    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/895d8ed4-6421-4018-a630-fb995aadf0bc-host-slash\") pod \"iptables-alerter-24lkk\" (UID: \"895d8ed4-6421-4018-a630-fb995aadf0bc\") " pod="openshift-network-operator/iptables-alerter-24lkk"
Oct 30 02:42:00.618462 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618432    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-var-lib-kubelet\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.618599 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618473    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-default-certificate\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:00.618599 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618533    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49a36e47-9ab6-49f4-bde8-85b78ba7c208-cni-binary-copy\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:42:00.618599 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618546    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d6516df6-4df4-4213-890a-0b25e274e606-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:42:00.618599 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618515    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-ovn-node-metrics-cert\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.618599 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618579    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d28e48a8-f70c-4949-b407-4c0e632f6a35-agent-certs\") pod \"konnectivity-agent-9xpbx\" (UID: \"d28e48a8-f70c-4949-b407-4c0e632f6a35\") " pod="kube-system/konnectivity-agent-9xpbx"
Oct 30 02:42:00.618809 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618617    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:42:00.618809 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618637    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70f081e1-7202-4b86-930d-8c3f72a3bb7e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jwzx9\" (UID: \"70f081e1-7202-4b86-930d-8c3f72a3bb7e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:42:00.618809 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618668    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-ovnkube-config\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.618809 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618735    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.618809 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618769    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-default-certificate\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:00.618809 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618781    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-installation-pull-secrets\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:00.619003 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618829    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-ovnkube-script-lib\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.619003 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618868    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70f081e1-7202-4b86-930d-8c3f72a3bb7e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jwzx9\" (UID: \"70f081e1-7202-4b86-930d-8c3f72a3bb7e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:42:00.619003 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618875    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49a36e47-9ab6-49f4-bde8-85b78ba7c208-cni-binary-copy\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:42:00.619003 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618877    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-ovnkube-config\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.619003 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618870    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56c4a0de-1d41-4b83-9bac-ddee3f43879b-cni-binary-copy\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.619003 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618927    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:42:00.619003 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618951    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxvfq\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-kube-api-access-dxvfq\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:00.619003 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618979    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc5jg\" (UniqueName: \"kubernetes.io/projected/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8-kube-api-access-dc5jg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg\" (UID: \"dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"
Oct 30 02:42:00.619003 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.618995    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d28e48a8-f70c-4949-b407-4c0e632f6a35-agent-certs\") pod \"konnectivity-agent-9xpbx\" (UID: \"d28e48a8-f70c-4949-b407-4c0e632f6a35\") " pod="kube-system/konnectivity-agent-9xpbx"
Oct 30 02:42:00.619387 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619019    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-ovnkube-script-lib\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.619387 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619062    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:42:00.619387 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619066    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-installation-pull-secrets\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:00.619387 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619099    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-sys-fs\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.619387 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619149    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56c4a0de-1d41-4b83-9bac-ddee3f43879b-cni-binary-copy\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.619387 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619160    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-bound-sa-token\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:00.619387 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619220    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-service-ca-bundle\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:42:00.619387 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619249    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c947k\" (UniqueName: \"kubernetes.io/projected/dfa424b6-593b-44ac-8e28-c9ba0e3207e8-kube-api-access-c947k\") pod \"network-check-source-6f78d59f99-7hpk4\" (UID: \"dfa424b6-593b-44ac-8e28-c9ba0e3207e8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hpk4"
Oct 30 02:42:00.619387 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619278    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-device-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.619387 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619302    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f697930c-fefc-42f2-835e-7b41cf115aaf-cert\") pod \"ingress-canary-fx6bk\" (UID: \"f697930c-fefc-42f2-835e-7b41cf115aaf\") " pod="openshift-ingress-canary/ingress-canary-fx6bk"
Oct 30 02:42:00.619387 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619328    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j6pt\" (UniqueName: \"kubernetes.io/projected/b1337da2-13ee-4b79-9a84-37ef217ae10f-kube-api-access-6j6pt\") pod \"kas-connection-checker-685895c7c9-2sq5m\" (UID: \"b1337da2-13ee-4b79-9a84-37ef217ae10f\") " pod="kube-system/kas-connection-checker-685895c7c9-2sq5m"
Oct 30 02:42:00.619387 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619358    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-hostroot\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.619387 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619374    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-service-ca-bundle\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:42:00.619387 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619383    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-conf-dir\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.619829 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619438    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwzt\" (UniqueName: \"kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt\") pod \"network-check-target-wx7bz\" (UID: \"d3d2494a-0ef0-4da6-b6dd-225f750bb909\") " pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:42:00.619829 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619473    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:00.619829 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619492    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-snapshots\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:42:00.619829 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619519    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-cni-dir\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.619829 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619543    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxg9s\" (UniqueName: \"kubernetes.io/projected/06715a40-fb77-4b9c-89e1-0aefe7b12a72-kube-api-access-jxg9s\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:42:00.619829 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619547    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f697930c-fefc-42f2-835e-7b41cf115aaf-cert\") pod \"ingress-canary-fx6bk\" (UID: \"f697930c-fefc-42f2-835e-7b41cf115aaf\") " pod="openshift-ingress-canary/ingress-canary-fx6bk"
Oct 30 02:42:00.619829 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619562    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-slash\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.619829 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619597    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-cni-bin\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.619829 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619608    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-snapshots\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:42:00.619829 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619695    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-var-lib-openvswitch\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.619829 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619739    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49a36e47-9ab6-49f4-bde8-85b78ba7c208-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:42:00.619829 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619776    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jwzx9\" (UID: \"70f081e1-7202-4b86-930d-8c3f72a3bb7e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:42:00.619829 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619799    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-run-systemd\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.619829 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619830    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-log-socket\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.620287 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619859    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kfst\" (UniqueName: \"kubernetes.io/projected/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-kube-api-access-4kfst\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:42:00.620287 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619898    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-sysctl-d\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.620287 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619941    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-sysctl-conf\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.620287 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.619982    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49a36e47-9ab6-49f4-bde8-85b78ba7c208-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:42:00.620287 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620019    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:42:00.620287 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620114    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs9h4\" (UniqueName: \"kubernetes.io/projected/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-api-access-bs9h4\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:00.620287 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620127    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49a36e47-9ab6-49f4-bde8-85b78ba7c208-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:42:00.620287 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620147    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-ca-trust-extracted\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:00.620287 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620201    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-trusted-ca\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:00.620287 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620249    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-ca-trust-extracted\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:00.620287 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620278    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-modprobe-d\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.620748 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620375    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-kubernetes\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.620748 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620397    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtmxt\" (UniqueName: \"kubernetes.io/projected/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-kube-api-access-vtmxt\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:42:00.620748 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620414    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb0c5682-acc6-4029-831d-b6aa9027db25-service-ca-bundle\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:00.620748 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620436    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-trusted-ca\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:00.620748 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620451    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e84dd1-8ee0-4a74-8802-3e3200c96642-tmp-dir\") pod \"node-resolver-6pzrk\" (UID: \"38e84dd1-8ee0-4a74-8802-3e3200c96642\") " pod="openshift-dns/node-resolver-6pzrk"
Oct 30 02:42:00.620748 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620467    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpt6g\" (UniqueName: \"kubernetes.io/projected/75211257-e617-40de-bf22-878d736448d1-kube-api-access-lpt6g\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.620748 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620483    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7-host\") pod \"node-ca-tvbs5\" (UID: \"a2796ed5-6fd6-4231-9847-1bee8c9f9dd7\") " pod="openshift-image-registry/node-ca-tvbs5"
Oct 30 02:42:00.620748 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620503    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kh57\" (UniqueName: \"kubernetes.io/projected/895d8ed4-6421-4018-a630-fb995aadf0bc-kube-api-access-6kh57\") pod \"iptables-alerter-24lkk\" (UID: \"895d8ed4-6421-4018-a630-fb995aadf0bc\") " pod="openshift-network-operator/iptables-alerter-24lkk"
Oct 30 02:42:00.620748 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620515    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-trusted-ca\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:00.620748 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620528    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-tuned\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.620748 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620570    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d49cl\" (UniqueName: \"kubernetes.io/projected/d6516df6-4df4-4213-890a-0b25e274e606-kube-api-access-d49cl\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:42:00.620748 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620604    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e84dd1-8ee0-4a74-8802-3e3200c96642-tmp-dir\") pod \"node-resolver-6pzrk\" (UID: \"38e84dd1-8ee0-4a74-8802-3e3200c96642\") " pod="openshift-dns/node-resolver-6pzrk"
Oct 30 02:42:00.620748 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620611    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqbmt\" (UniqueName: \"kubernetes.io/projected/60924e48-a5e3-4f2b-ab35-48479c3b730b-kube-api-access-lqbmt\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:42:00.620748 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620608    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-tuned\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.620748 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620645    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d28e48a8-f70c-4949-b407-4c0e632f6a35-konnectivity-ca\") pod \"konnectivity-agent-9xpbx\" (UID: \"d28e48a8-f70c-4949-b407-4c0e632f6a35\") " pod="kube-system/konnectivity-agent-9xpbx"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620768    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-cni-netd\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620803    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-trusted-ca\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620870    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b93ba591-108f-41bd-afed-012dcd2f8690-ca-trust-extracted\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620907    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620919    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b93ba591-108f-41bd-afed-012dcd2f8690-ca-trust-extracted\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620936    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d28e48a8-f70c-4949-b407-4c0e632f6a35-konnectivity-ca\") pod \"konnectivity-agent-9xpbx\" (UID: \"d28e48a8-f70c-4949-b407-4c0e632f6a35\") " pod="kube-system/konnectivity-agent-9xpbx"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620938    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-tmp\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.620977    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621000    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-registry-certificates\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621002    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-tmp\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621021    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-tmp-dir\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621040    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smpsr\" (UniqueName: \"kubernetes.io/projected/f697930c-fefc-42f2-835e-7b41cf115aaf-kube-api-access-smpsr\") pod \"ingress-canary-fx6bk\" (UID: \"f697930c-fefc-42f2-835e-7b41cf115aaf\") " pod="openshift-ingress-canary/ingress-canary-fx6bk"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621054    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-run\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621075    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-host\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621091    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-system-cni-dir\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621106    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-etc-kubernetes\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621123    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-bound-sa-token\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621106    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-tmp-dir\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621149    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpm79\" (UniqueName: \"kubernetes.io/projected/0d2d9d6b-2705-49d3-b3d2-079d6d85148f-kube-api-access-tpm79\") pod \"kas-connection-checker-685895c7c9-jsbwd\" (UID: \"0d2d9d6b-2705-49d3-b3d2-079d6d85148f\") " pod="kube-system/kas-connection-checker-685895c7c9-jsbwd"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621241    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-registry-certificates\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621339    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-socket-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621360    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:42:00.621411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621376    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-var-lib-kubelet\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621505    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49a36e47-9ab6-49f4-bde8-85b78ba7c208-cnibin\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621553    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3ebba8-b807-4256-b552-5a18ec8412cb-original-pull-secret\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621564    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49a36e47-9ab6-49f4-bde8-85b78ba7c208-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621668    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svt6f\" (UniqueName: \"kubernetes.io/projected/88d453b9-7e71-4561-8e1b-4b4074651675-kube-api-access-svt6f\") pod \"service-ca-operator-544d749c8c-vbz2s\" (UID: \"88d453b9-7e71-4561-8e1b-4b4074651675\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621688    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-image-registry-private-configuration\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621705    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-installation-pull-secrets\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621720    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-trusted-ca\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621736    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg\" (UID: \"dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621760    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-run-netns\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621785    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbq75\" (UniqueName: \"kubernetes.io/projected/cb0c5682-acc6-4029-831d-b6aa9027db25-kube-api-access-cbq75\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621776    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49a36e47-9ab6-49f4-bde8-85b78ba7c208-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621816    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-env-overrides\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621855    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-data-volume\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621889    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg\" (UID: \"dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621901    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88d453b9-7e71-4561-8e1b-4b4074651675-serving-cert\") pod \"service-ca-operator-544d749c8c-vbz2s\" (UID: \"88d453b9-7e71-4561-8e1b-4b4074651675\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621922    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-image-registry-private-configuration\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621929    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-installation-pull-secrets\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621935    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-registration-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621970    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-env-overrides\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621980    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-data-volume\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.621984    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49a36e47-9ab6-49f4-bde8-85b78ba7c208-system-cni-dir\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622025    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-trusted-ca\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622034    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88d453b9-7e71-4561-8e1b-4b4074651675-serving-cert\") pod \"service-ca-operator-544d749c8c-vbz2s\" (UID: \"88d453b9-7e71-4561-8e1b-4b4074651675\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622049    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-systemd-units\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622069    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-var-lib-cni-bin\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622099    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-config-volume\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622138    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-run-openvswitch\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622222    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmg6l\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-kube-api-access-hmg6l\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622244    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-serving-cert\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622261    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsrlt\" (UniqueName: \"kubernetes.io/projected/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-kube-api-access-tsrlt\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622281    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a3ebba8-b807-4256-b552-5a18ec8412cb-dbus\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622299    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-config-volume\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622308    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-systemd\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.622324 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622346    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-run-netns\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622380    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-serving-cert\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622381    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-crio-socket\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622422    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-image-registry-private-configuration\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622541    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-socket-dir-parent\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622558    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-image-registry-private-configuration\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622566    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-daemon-config\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622597    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-run-ovn\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622651    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622686    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl9cq\" (UniqueName: \"kubernetes.io/projected/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-kube-api-access-sl9cq\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622740    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-daemon-config\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622778    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622795    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-sysconfig\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622812    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7-serviceca\") pod \"node-ca-tvbs5\" (UID: \"a2796ed5-6fd6-4231-9847-1bee8c9f9dd7\") " pod="openshift-image-registry/node-ca-tvbs5"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622827    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpf67\" (UniqueName: \"kubernetes.io/projected/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7-kube-api-access-lpf67\") pod \"node-ca-tvbs5\" (UID: \"a2796ed5-6fd6-4231-9847-1bee8c9f9dd7\") " pod="openshift-image-registry/node-ca-tvbs5"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622842    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-var-lib-cni-multus\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622886    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-run-multus-certs\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622928    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-etc-openvswitch\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.622978    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-config\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.623036    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7-serviceca\") pod \"node-ca-tvbs5\" (UID: \"a2796ed5-6fd6-4231-9847-1bee8c9f9dd7\") " pod="openshift-image-registry/node-ca-tvbs5"
Oct 30 02:42:00.623454 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.623259    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-config\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:42:00.636993 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.636971    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl6q6\" (UniqueName: \"kubernetes.io/projected/7e8de64a-fd25-4aa6-9160-b25e09d116aa-kube-api-access-wl6q6\") pod \"volume-data-source-validator-58c9f5f46d-njw6d\" (UID: \"7e8de64a-fd25-4aa6-9160-b25e09d116aa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-njw6d"
Oct 30 02:42:00.638037 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.637841    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sv8m\" (UniqueName: \"kubernetes.io/projected/49a36e47-9ab6-49f4-bde8-85b78ba7c208-kube-api-access-6sv8m\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:42:00.638037 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.637847    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs8x4\" (UniqueName: \"kubernetes.io/projected/c1cf080d-9f7d-4614-b446-76d174d9c3f7-kube-api-access-xs8x4\") pod \"kas-connection-checker-685895c7c9-dhf78\" (UID: \"c1cf080d-9f7d-4614-b446-76d174d9c3f7\") " pod="kube-system/kas-connection-checker-685895c7c9-dhf78"
Oct 30 02:42:00.638037 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.637849    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cshhd\" (UniqueName: \"kubernetes.io/projected/38e84dd1-8ee0-4a74-8802-3e3200c96642-kube-api-access-cshhd\") pod \"node-resolver-6pzrk\" (UID: \"38e84dd1-8ee0-4a74-8802-3e3200c96642\") " pod="openshift-dns/node-resolver-6pzrk"
Oct 30 02:42:00.638037 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.637936    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnqw2\" (UniqueName: \"kubernetes.io/projected/56c4a0de-1d41-4b83-9bac-ddee3f43879b-kube-api-access-jnqw2\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.638037 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.637987    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm4tl\" (UniqueName: \"kubernetes.io/projected/cd6d2859-2141-401c-825d-b6f12bb72644-kube-api-access-tm4tl\") pod \"migrator-66df5d496b-dgtzw\" (UID: \"cd6d2859-2141-401c-825d-b6f12bb72644\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dgtzw"
Oct 30 02:42:00.638037 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.637995    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgfgs\" (UniqueName: \"kubernetes.io/projected/ca9b24b8-5841-4f54-aa38-518bfbb701c6-kube-api-access-vgfgs\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.641371 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.638702    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j6pt\" (UniqueName: \"kubernetes.io/projected/b1337da2-13ee-4b79-9a84-37ef217ae10f-kube-api-access-6j6pt\") pod \"kas-connection-checker-685895c7c9-2sq5m\" (UID: \"b1337da2-13ee-4b79-9a84-37ef217ae10f\") " pod="kube-system/kas-connection-checker-685895c7c9-2sq5m"
Oct 30 02:42:00.641371 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.639001    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc5jg\" (UniqueName: \"kubernetes.io/projected/dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8-kube-api-access-dc5jg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg\" (UID: \"dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg"
Oct 30 02:42:00.641371 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.639112    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c947k\" (UniqueName: \"kubernetes.io/projected/dfa424b6-593b-44ac-8e28-c9ba0e3207e8-kube-api-access-c947k\") pod \"network-check-source-6f78d59f99-7hpk4\" (UID: \"dfa424b6-593b-44ac-8e28-c9ba0e3207e8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7hpk4"
Oct 30 02:42:00.641371 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.639463    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkwzt\" (UniqueName: \"kubernetes.io/projected/d3d2494a-0ef0-4da6-b6dd-225f750bb909-kube-api-access-mkwzt\") pod \"network-check-target-wx7bz\" (UID: \"d3d2494a-0ef0-4da6-b6dd-225f750bb909\") " pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:42:00.644135 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.644114    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-bound-sa-token\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:00.644264 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.644237    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxvfq\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-kube-api-access-dxvfq\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:00.645287 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.645265    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpt6g\" (UniqueName: \"kubernetes.io/projected/75211257-e617-40de-bf22-878d736448d1-kube-api-access-lpt6g\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.645436 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.645419    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpf67\" (UniqueName: \"kubernetes.io/projected/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7-kube-api-access-lpf67\") pod \"node-ca-tvbs5\" (UID: \"a2796ed5-6fd6-4231-9847-1bee8c9f9dd7\") " pod="openshift-image-registry/node-ca-tvbs5"
Oct 30 02:42:00.645476 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.645453    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs9h4\" (UniqueName: \"kubernetes.io/projected/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-api-access-bs9h4\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:00.645651 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.645634    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kfst\" (UniqueName: \"kubernetes.io/projected/fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e-kube-api-access-4kfst\") pod \"console-operator-7cd56b8dbc-scstg\" (UID: \"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:42:00.646928 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.646895    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqbmt\" (UniqueName: \"kubernetes.io/projected/60924e48-a5e3-4f2b-ab35-48479c3b730b-kube-api-access-lqbmt\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:42:00.647036 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.646937    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl9cq\" (UniqueName: \"kubernetes.io/projected/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-kube-api-access-sl9cq\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.647036 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.646962    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kh57\" (UniqueName: \"kubernetes.io/projected/895d8ed4-6421-4018-a630-fb995aadf0bc-kube-api-access-6kh57\") pod \"iptables-alerter-24lkk\" (UID: \"895d8ed4-6421-4018-a630-fb995aadf0bc\") " pod="openshift-network-operator/iptables-alerter-24lkk"
Oct 30 02:42:00.647166 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.647144    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d49cl\" (UniqueName: \"kubernetes.io/projected/d6516df6-4df4-4213-890a-0b25e274e606-kube-api-access-d49cl\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:42:00.647221 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.647162    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxg9s\" (UniqueName: \"kubernetes.io/projected/06715a40-fb77-4b9c-89e1-0aefe7b12a72-kube-api-access-jxg9s\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:42:00.647384 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.647364    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtmxt\" (UniqueName: \"kubernetes.io/projected/f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9-kube-api-access-vtmxt\") pod \"insights-operator-7c4986d449-gxb72\" (UID: \"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9\") " pod="openshift-insights/insights-operator-7c4986d449-gxb72"
Oct 30 02:42:00.647657 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.647638    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsrlt\" (UniqueName: \"kubernetes.io/projected/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-kube-api-access-tsrlt\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:42:00.647699 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.647641    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmg6l\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-kube-api-access-hmg6l\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:00.647796 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.647770    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbq75\" (UniqueName: \"kubernetes.io/projected/cb0c5682-acc6-4029-831d-b6aa9027db25-kube-api-access-cbq75\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:00.647881 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.647771    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpm79\" (UniqueName: \"kubernetes.io/projected/0d2d9d6b-2705-49d3-b3d2-079d6d85148f-kube-api-access-tpm79\") pod \"kas-connection-checker-685895c7c9-jsbwd\" (UID: \"0d2d9d6b-2705-49d3-b3d2-079d6d85148f\") " pod="kube-system/kas-connection-checker-685895c7c9-jsbwd"
Oct 30 02:42:00.648500 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.648473    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svt6f\" (UniqueName: \"kubernetes.io/projected/88d453b9-7e71-4561-8e1b-4b4074651675-kube-api-access-svt6f\") pod \"service-ca-operator-544d749c8c-vbz2s\" (UID: \"88d453b9-7e71-4561-8e1b-4b4074651675\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s"
Oct 30 02:42:00.648572 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.648557    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smpsr\" (UniqueName: \"kubernetes.io/projected/f697930c-fefc-42f2-835e-7b41cf115aaf-kube-api-access-smpsr\") pod \"ingress-canary-fx6bk\" (UID: \"f697930c-fefc-42f2-835e-7b41cf115aaf\") " pod="openshift-ingress-canary/ingress-canary-fx6bk"
Oct 30 02:42:00.648606 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.648581    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-bound-sa-token\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:00.709535 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.709465    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:00.723519 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723500    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-sys-fs\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.723582 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723526    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-device-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.723582 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723542    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-hostroot\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.723582 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723556    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-conf-dir\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.723582 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723572    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:00.723772 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723586    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-cni-dir\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.723772 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723600    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-slash\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.723772 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723626    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-hostroot\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.723772 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723636    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-sys-fs\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.723772 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723626    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-device-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.723772 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723639    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-slash\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.723772 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723655    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-cni-bin\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.723772 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723658    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-conf-dir\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.723772 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723682    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-var-lib-openvswitch\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.723772 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723687    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-cni-bin\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.723772 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723682    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-cni-dir\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.723772 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723705    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49a36e47-9ab6-49f4-bde8-85b78ba7c208-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:42:00.723772 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.723713    4960 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:42:00.723772 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723725    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-var-lib-openvswitch\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.723772 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723729    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jwzx9\" (UID: \"70f081e1-7202-4b86-930d-8c3f72a3bb7e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:42:00.723772 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723763    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-run-systemd\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.723772 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723776    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49a36e47-9ab6-49f4-bde8-85b78ba7c208-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723789    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-run-systemd\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.723792    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls podName:6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da nodeName:}" failed. No retries permitted until 2026-10-30 02:42:01.223770938 +0000 UTC m=+1.872819358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dkfnj" (UID: "6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.723793    4960 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723818    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-log-socket\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723838    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-sysctl-d\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.723846    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert podName:70f081e1-7202-4b86-930d-8c3f72a3bb7e nodeName:}" failed. No retries permitted until 2026-10-30 02:42:01.223835295 +0000 UTC m=+1.872883698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jwzx9" (UID: "70f081e1-7202-4b86-930d-8c3f72a3bb7e") : secret "networking-console-plugin-cert" not found
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723850    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-log-socket\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723870    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-sysctl-conf\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723899    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723923    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-sysctl-d\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723925    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-modprobe-d\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723953    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-kubernetes\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723969    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb0c5682-acc6-4029-831d-b6aa9027db25-service-ca-bundle\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.723984    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7-host\") pod \"node-ca-tvbs5\" (UID: \"a2796ed5-6fd6-4231-9847-1bee8c9f9dd7\") " pod="openshift-image-registry/node-ca-tvbs5"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724000    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-cni-netd\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724002    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-modprobe-d\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724015    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-sysctl-conf\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.724017    4960 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724018    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724049    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-cni-netd\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724057    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.724070    4960 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724070    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-kubernetes\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.724087    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls podName:06715a40-fb77-4b9c-89e1-0aefe7b12a72 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:01.224070259 +0000 UTC m=+1.873118673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7mfqp" (UID: "06715a40-fb77-4b9c-89e1-0aefe7b12a72") : secret "samples-operator-tls" not found
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724109    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2796ed5-6fd6-4231-9847-1bee8c9f9dd7-host\") pod \"node-ca-tvbs5\" (UID: \"a2796ed5-6fd6-4231-9847-1bee8c9f9dd7\") " pod="openshift-image-registry/node-ca-tvbs5"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724113    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-run\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724087    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.724147    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls podName:c5d42a36-81a2-49ad-9fa0-32d42e87d75d nodeName:}" failed. No retries permitted until 2026-10-30 02:42:01.224100544 +0000 UTC m=+1.873148945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls") pod "dns-default-7mhtz" (UID: "c5d42a36-81a2-49ad-9fa0-32d42e87d75d") : secret "dns-default-metrics-tls" not found
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724168    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-run\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724171    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-host\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724223    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-system-cni-dir\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724255    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-system-cni-dir\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724257    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-etc-kubernetes\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724273    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-host\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724289    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-socket-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724312    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-etc-kubernetes\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724315    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-var-lib-kubelet\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724342    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49a36e47-9ab6-49f4-bde8-85b78ba7c208-cnibin\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724371    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-run-netns\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724374    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-var-lib-kubelet\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724346    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-socket-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724416    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-run-netns\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724412    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-registration-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724417    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49a36e47-9ab6-49f4-bde8-85b78ba7c208-cnibin\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724451    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49a36e47-9ab6-49f4-bde8-85b78ba7c208-system-cni-dir\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724455    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-registration-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724475    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-systemd-units\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724512    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-systemd-units\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724516    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49a36e47-9ab6-49f4-bde8-85b78ba7c208-system-cni-dir\") pod \"multus-additional-cni-plugins-mjph2\" (UID: \"49a36e47-9ab6-49f4-bde8-85b78ba7c208\") " pod="openshift-multus/multus-additional-cni-plugins-mjph2"
Oct 30 02:42:00.724501 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724502    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-var-lib-cni-bin\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724550    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-run-openvswitch\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724579    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-var-lib-cni-bin\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724589    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a3ebba8-b807-4256-b552-5a18ec8412cb-dbus\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724612    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-systemd\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724608    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-run-openvswitch\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724636    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-run-netns\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724651    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a3ebba8-b807-4256-b552-5a18ec8412cb-dbus\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724660    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-crio-socket\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724676    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb0c5682-acc6-4029-831d-b6aa9027db25-service-ca-bundle\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724682    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-run-netns\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724685    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-socket-dir-parent\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724664    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-systemd\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724709    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-crio-socket\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724724    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-run-ovn\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724742    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-multus-socket-dir-parent\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724749    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-run-ovn\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724750    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724778    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724783    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724817    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-sysconfig\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.724832    4960 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724843    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-var-lib-cni-multus\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724867    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-run-multus-certs\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.724887    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm podName:6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da nodeName:}" failed. No retries permitted until 2026-10-30 02:42:01.224871778 +0000 UTC m=+1.873920196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dkfnj" (UID: "6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da") : configmap "kube-rbac-proxy" not found
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724897    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-etc-sysconfig\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724901    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-run-multus-certs\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724900    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-var-lib-cni-multus\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724910    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-etc-openvswitch\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724934    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-kubelet\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724940    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-etc-openvswitch\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724949    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724969    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724984    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-host-kubelet\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.724992    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-lib-modules\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725012    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725034    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a3ebba8-b807-4256-b552-5a18ec8412cb-kubelet-config\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.725043    4960 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.725052    4960 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59bd5b5c6b-lcbtn: secret "image-registry-tls" not found
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725054    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-cnibin\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725080    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-run-k8s-cni-cncf-io\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725099    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-node-log\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.725111    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls podName:bedee86a-79f7-43cf-9ed2-5dbf5b8109eb nodeName:}" failed. No retries permitted until 2026-10-30 02:42:01.225100201 +0000 UTC m=+1.874148626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls") pod "image-registry-59bd5b5c6b-lcbtn" (UID: "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb") : secret "image-registry-tls" not found
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725111    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-lib-modules\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725138    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c022b3d-db3f-45a7-b6b7-5404c6f170c8-node-log\") pod \"ovnkube-node-fl65k\" (UID: \"3c022b3d-db3f-45a7-b6b7-5404c6f170c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725145    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-cnibin\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725149    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/56c4a0de-1d41-4b83-9bac-ddee3f43879b-host-run-k8s-cni-cncf-io\") pod \"multus-kbn25\" (UID: \"56c4a0de-1d41-4b83-9bac-ddee3f43879b\") " pod="openshift-multus/multus-kbn25"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725168    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a3ebba8-b807-4256-b552-5a18ec8412cb-kubelet-config\") pod \"global-pull-secret-syncer-skc8q\" (UID: \"8a3ebba8-b807-4256-b552-5a18ec8412cb\") " pod="kube-system/global-pull-secret-syncer-skc8q"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725192    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-sys\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725225    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725228    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-sys\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725249    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725266    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/38e84dd1-8ee0-4a74-8802-3e3200c96642-hosts-file\") pod \"node-resolver-6pzrk\" (UID: \"38e84dd1-8ee0-4a74-8802-3e3200c96642\") " pod="openshift-dns/node-resolver-6pzrk"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725279    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/895d8ed4-6421-4018-a630-fb995aadf0bc-host-slash\") pod \"iptables-alerter-24lkk\" (UID: \"895d8ed4-6421-4018-a630-fb995aadf0bc\") " pod="openshift-network-operator/iptables-alerter-24lkk"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725297    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-var-lib-kubelet\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725314    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.725321    4960 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.725325    4960 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.725331    4960 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c6dbd69f-lrwtw: secret "image-registry-tls" not found
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725335    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725346    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/895d8ed4-6421-4018-a630-fb995aadf0bc-host-slash\") pod \"iptables-alerter-24lkk\" (UID: \"895d8ed4-6421-4018-a630-fb995aadf0bc\") " pod="openshift-network-operator/iptables-alerter-24lkk"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725364    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.725373    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs podName:cb0c5682-acc6-4029-831d-b6aa9027db25 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:01.225359304 +0000 UTC m=+1.874407729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs") pod "router-default-5d4bdb4f7f-89p4x" (UID: "cb0c5682-acc6-4029-831d-b6aa9027db25") : secret "router-metrics-certs-default" not found
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725378    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/38e84dd1-8ee0-4a74-8802-3e3200c96642-hosts-file\") pod \"node-resolver-6pzrk\" (UID: \"38e84dd1-8ee0-4a74-8802-3e3200c96642\") " pod="openshift-dns/node-resolver-6pzrk"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.725392    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls podName:b93ba591-108f-41bd-afed-012dcd2f8690 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:01.225382241 +0000 UTC m=+1.874430646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls") pod "image-registry-7c6dbd69f-lrwtw" (UID: "b93ba591-108f-41bd-afed-012dcd2f8690") : secret "image-registry-tls" not found
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725393    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca9b24b8-5841-4f54-aa38-518bfbb701c6-var-lib-kubelet\") pod \"tuned-vx9zb\" (UID: \"ca9b24b8-5841-4f54-aa38-518bfbb701c6\") " pod="openshift-cluster-node-tuning-operator/tuned-vx9zb"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.725419    4960 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.725427    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/75211257-e617-40de-bf22-878d736448d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-kk76t\" (UID: \"75211257-e617-40de-bf22-878d736448d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kk76t"
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.725435    4960 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.725447    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls podName:d6516df6-4df4-4213-890a-0b25e274e606 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:01.225437966 +0000 UTC m=+1.874486365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vjvcf" (UID: "d6516df6-4df4-4213-890a-0b25e274e606") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:42:00.726145 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:00.725472    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs podName:60924e48-a5e3-4f2b-ab35-48479c3b730b nodeName:}" failed. No retries permitted until 2026-10-30 02:42:01.225463342 +0000 UTC m=+1.874511741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs") pod "network-metrics-daemon-t5747" (UID: "60924e48-a5e3-4f2b-ab35-48479c3b730b") : secret "metrics-daemon-secret" not found
Oct 30 02:42:00.822986 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.822966    4960 scope.go:122] "RemoveContainer" containerID="d49244d3852bbacd675ebd114cf975faac54dc9d1ec0822431ad05f17f237c73"
Oct 30 02:42:00.838967 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:00.838948    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:01.004267 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.004247    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:01.069779 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.069757    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6pzrk_38e84dd1-8ee0-4a74-8802-3e3200c96642/dns-node-resolver/0.log"
Oct 30 02:42:01.229020 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.228985    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:42:01.229150 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.229037    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:01.229150 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.229063    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:01.229150 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.229087    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:01.229150 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.229102    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:01.229150 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229120    4960 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:42:01.229392 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229171    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls podName:c5d42a36-81a2-49ad-9fa0-32d42e87d75d nodeName:}" failed. No retries permitted until 2026-10-30 02:42:02.229156779 +0000 UTC m=+2.878205183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls") pod "dns-default-7mhtz" (UID: "c5d42a36-81a2-49ad-9fa0-32d42e87d75d") : secret "dns-default-metrics-tls" not found
Oct 30 02:42:01.229392 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.229124    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:42:01.229392 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229211    4960 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:42:01.229392 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229220    4960 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:42:01.229392 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229233    4960 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:42:01.229392 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229246    4960 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c6dbd69f-lrwtw: secret "image-registry-tls" not found
Oct 30 02:42:01.229392 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229251    4960 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:42:01.229392 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.229263    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:42:01.229392 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229271    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs podName:cb0c5682-acc6-4029-831d-b6aa9027db25 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:02.229257452 +0000 UTC m=+2.878305869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs") pod "router-default-5d4bdb4f7f-89p4x" (UID: "cb0c5682-acc6-4029-831d-b6aa9027db25") : secret "router-metrics-certs-default" not found
Oct 30 02:42:01.229392 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229225    4960 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59bd5b5c6b-lcbtn: secret "image-registry-tls" not found
Oct 30 02:42:01.229392 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229277    4960 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:42:01.229392 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229298    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls podName:b93ba591-108f-41bd-afed-012dcd2f8690 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:02.229283176 +0000 UTC m=+2.878331583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls") pod "image-registry-7c6dbd69f-lrwtw" (UID: "b93ba591-108f-41bd-afed-012dcd2f8690") : secret "image-registry-tls" not found
Oct 30 02:42:01.229392 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229316    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm podName:6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da nodeName:}" failed. No retries permitted until 2026-10-30 02:42:02.229309165 +0000 UTC m=+2.878357564 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dkfnj" (UID: "6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da") : configmap "kube-rbac-proxy" not found
Oct 30 02:42:01.229392 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229331    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls podName:bedee86a-79f7-43cf-9ed2-5dbf5b8109eb nodeName:}" failed. No retries permitted until 2026-10-30 02:42:02.229324344 +0000 UTC m=+2.878372743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls") pod "image-registry-59bd5b5c6b-lcbtn" (UID: "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb") : secret "image-registry-tls" not found
Oct 30 02:42:01.229392 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229345    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls podName:d6516df6-4df4-4213-890a-0b25e274e606 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:02.229337932 +0000 UTC m=+2.878386330 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vjvcf" (UID: "d6516df6-4df4-4213-890a-0b25e274e606") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:42:01.229392 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229343    4960 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:42:01.229392 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.229380    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:01.229392 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229393    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs podName:60924e48-a5e3-4f2b-ab35-48479c3b730b nodeName:}" failed. No retries permitted until 2026-10-30 02:42:02.22937984 +0000 UTC m=+2.878428257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs") pod "network-metrics-daemon-t5747" (UID: "60924e48-a5e3-4f2b-ab35-48479c3b730b") : secret "metrics-daemon-secret" not found
Oct 30 02:42:01.229934 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229431    4960 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:42:01.229934 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.229432    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jwzx9\" (UID: \"70f081e1-7202-4b86-930d-8c3f72a3bb7e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:42:01.229934 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229462    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls podName:6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da nodeName:}" failed. No retries permitted until 2026-10-30 02:42:02.229451976 +0000 UTC m=+2.878500383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dkfnj" (UID: "6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:42:01.229934 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229489    4960 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:42:01.229934 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.229488    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:42:01.229934 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229516    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert podName:70f081e1-7202-4b86-930d-8c3f72a3bb7e nodeName:}" failed. No retries permitted until 2026-10-30 02:42:02.229506404 +0000 UTC m=+2.878554818 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jwzx9" (UID: "70f081e1-7202-4b86-930d-8c3f72a3bb7e") : secret "networking-console-plugin-cert" not found
Oct 30 02:42:01.229934 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229583    4960 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:42:01.229934 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.229628    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls podName:06715a40-fb77-4b9c-89e1-0aefe7b12a72 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:02.229617752 +0000 UTC m=+2.878666170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7mfqp" (UID: "06715a40-fb77-4b9c-89e1-0aefe7b12a72") : secret "samples-operator-tls" not found
Oct 30 02:42:01.464470 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.464440    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:01.560386 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.560366    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 02:42:01.560843 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.560829    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/1.log"
Oct 30 02:42:01.560899 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.560858    4960 generic.go:356] "Generic (PLEG): container finished" podID="fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e" containerID="aa43f12ca5e4cb025b1bf59fcdf00f98211ba44c2888f10f9756b2b636d7c1a2" exitCode=255
Oct 30 02:42:01.560958 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.560932    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg" event={"ID":"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e","Type":"ContainerDied","Data":"aa43f12ca5e4cb025b1bf59fcdf00f98211ba44c2888f10f9756b2b636d7c1a2"}
Oct 30 02:42:01.561002 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.560985    4960 scope.go:122] "RemoveContainer" containerID="d49244d3852bbacd675ebd114cf975faac54dc9d1ec0822431ad05f17f237c73"
Oct 30 02:42:01.561737 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.561687    4960 scope.go:122] "RemoveContainer" containerID="aa43f12ca5e4cb025b1bf59fcdf00f98211ba44c2888f10f9756b2b636d7c1a2"
Oct 30 02:42:01.564261 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:01.561963    4960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-scstg_openshift-console-operator(fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg" podUID="fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e"
Oct 30 02:42:01.631906 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:01.631878    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:02.238843 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.238804    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:42:02.238843 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.238843    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:42:02.239093 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.238863    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:02.239093 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.238883    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jwzx9\" (UID: \"70f081e1-7202-4b86-930d-8c3f72a3bb7e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:42:02.239093 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.238901    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:42:02.239093 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.238925    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:42:02.239093 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:02.238964    4960 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:42:02.239093 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.238975    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:02.239093 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.239010    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:02.239093 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:02.238971    4960 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:42:02.239093 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:02.239035    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls podName:6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da nodeName:}" failed. No retries permitted until 2026-10-30 02:42:04.239015848 +0000 UTC m=+4.888064270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dkfnj" (UID: "6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:42:02.239093 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:02.239036    4960 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:42:02.239093 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:02.239103    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs podName:60924e48-a5e3-4f2b-ab35-48479c3b730b nodeName:}" failed. No retries permitted until 2026-10-30 02:42:04.239063992 +0000 UTC m=+4.888112422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs") pod "network-metrics-daemon-t5747" (UID: "60924e48-a5e3-4f2b-ab35-48479c3b730b") : secret "metrics-daemon-secret" not found
Oct 30 02:42:02.239630 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:02.238964    4960 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:42:02.239630 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:02.239151    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls podName:06715a40-fb77-4b9c-89e1-0aefe7b12a72 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:04.239139759 +0000 UTC m=+4.888188161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7mfqp" (UID: "06715a40-fb77-4b9c-89e1-0aefe7b12a72") : secret "samples-operator-tls" not found
Oct 30 02:42:02.239630 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.239128    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:02.239630 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:02.239010    4960 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:42:02.239630 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:02.239226    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls podName:d6516df6-4df4-4213-890a-0b25e274e606 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:04.2392035 +0000 UTC m=+4.888251916 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vjvcf" (UID: "d6516df6-4df4-4213-890a-0b25e274e606") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:42:02.239630 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:02.239015    4960 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:42:02.239630 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.239260    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:02.239630 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:02.239293    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls podName:c5d42a36-81a2-49ad-9fa0-32d42e87d75d nodeName:}" failed. No retries permitted until 2026-10-30 02:42:04.239277051 +0000 UTC m=+4.888325468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls") pod "dns-default-7mhtz" (UID: "c5d42a36-81a2-49ad-9fa0-32d42e87d75d") : secret "dns-default-metrics-tls" not found
Oct 30 02:42:02.239630 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:02.239318    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert podName:70f081e1-7202-4b86-930d-8c3f72a3bb7e nodeName:}" failed. No retries permitted until 2026-10-30 02:42:04.23930982 +0000 UTC m=+4.888358219 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jwzx9" (UID: "70f081e1-7202-4b86-930d-8c3f72a3bb7e") : secret "networking-console-plugin-cert" not found
Oct 30 02:42:02.239630 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.239329    4960 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:42:02.239630 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.239512    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:02.241840 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.241816    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb0c5682-acc6-4029-831d-b6aa9027db25-metrics-certs\") pod \"router-default-5d4bdb4f7f-89p4x\" (UID: \"cb0c5682-acc6-4029-831d-b6aa9027db25\") " pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:02.241941 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.241851    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls\") pod \"image-registry-7c6dbd69f-lrwtw\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") " pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:02.241941 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.241905    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls\") pod \"image-registry-59bd5b5c6b-lcbtn\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") " pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:02.245511 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.245480    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:02.321250 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.321230    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:02.327508 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.327492    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:02.333737 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.333720    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:02.413904 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.410780    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:02.464757 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.463752    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tvbs5_a2796ed5-6fd6-4231-9847-1bee8c9f9dd7/node-ca/0.log"
Oct 30 02:42:02.566663 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.566643    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 02:42:02.596986 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.596966    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:02.657061 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.657042    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5d4bdb4f7f-89p4x"]
Oct 30 02:42:02.658693 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:42:02.658655    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0c5682_acc6_4029_831d_b6aa9027db25.slice/crio-f7528e4a33e85cb892397209e0ff317362405feb0aa85bfc42281aea3aa5ff49 WatchSource:0}: Error finding container f7528e4a33e85cb892397209e0ff317362405feb0aa85bfc42281aea3aa5ff49: Status 404 returned error can't find the container with id f7528e4a33e85cb892397209e0ff317362405feb0aa85bfc42281aea3aa5ff49
Oct 30 02:42:02.691620 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.691599    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c6dbd69f-lrwtw"]
Oct 30 02:42:02.700012 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:42:02.699993    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93ba591_108f_41bd_afed_012dcd2f8690.slice/crio-f4514f7f51047f78e8144152ec61451caa464215133c5b0b41c99766df0713dd WatchSource:0}: Error finding container f4514f7f51047f78e8144152ec61451caa464215133c5b0b41c99766df0713dd: Status 404 returned error can't find the container with id f4514f7f51047f78e8144152ec61451caa464215133c5b0b41c99766df0713dd
Oct 30 02:42:02.717624 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.717607    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"]
Oct 30 02:42:02.719569 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:42:02.719547    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbedee86a_79f7_43cf_9ed2_5dbf5b8109eb.slice/crio-495895cd768c167eb52825da5a0731ccd66de7add9512ea57e3a767e77289d29 WatchSource:0}: Error finding container 495895cd768c167eb52825da5a0731ccd66de7add9512ea57e3a767e77289d29: Status 404 returned error can't find the container with id 495895cd768c167eb52825da5a0731ccd66de7add9512ea57e3a767e77289d29
Oct 30 02:42:02.802553 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.802526    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:02.978654 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:02.978569    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:03.064826 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:03.064794    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fx6bk_f697930c-fefc-42f2-835e-7b41cf115aaf/serve-healthcheck-canary/0.log"
Oct 30 02:42:03.113772 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:03.113745    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:03.220531 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:03.220506    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:03.263589 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:03.263541    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-dgtzw_cd6d2859-2141-401c-825d-b6f12bb72644/migrator/0.log"
Oct 30 02:42:03.395346 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:03.395316    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:42:03.395681 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:03.395660    4960 scope.go:122] "RemoveContainer" containerID="aa43f12ca5e4cb025b1bf59fcdf00f98211ba44c2888f10f9756b2b636d7c1a2"
Oct 30 02:42:03.395852 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:03.395834    4960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-scstg_openshift-console-operator(fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg" podUID="fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e"
Oct 30 02:42:03.571924 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:03.571887    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x" event={"ID":"cb0c5682-acc6-4029-831d-b6aa9027db25","Type":"ContainerStarted","Data":"ad958b82cb8911e7951093b84f15aca644191400c16e8eb2c784fd82ca2eefbb"}
Oct 30 02:42:03.572355 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:03.571937    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x" event={"ID":"cb0c5682-acc6-4029-831d-b6aa9027db25","Type":"ContainerStarted","Data":"f7528e4a33e85cb892397209e0ff317362405feb0aa85bfc42281aea3aa5ff49"}
Oct 30 02:42:03.573825 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:03.573795    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw" event={"ID":"b93ba591-108f-41bd-afed-012dcd2f8690","Type":"ContainerStarted","Data":"e4e46ff6b889767ca4e25938f9a118549873e78fa46895a7dafe2d44c6a8e02d"}
Oct 30 02:42:03.573939 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:03.573840    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw" event={"ID":"b93ba591-108f-41bd-afed-012dcd2f8690","Type":"ContainerStarted","Data":"f4514f7f51047f78e8144152ec61451caa464215133c5b0b41c99766df0713dd"}
Oct 30 02:42:03.575430 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:03.575407    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn" event={"ID":"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb","Type":"ContainerStarted","Data":"d9856c58531e30fcc09dfd51ef493836c0303a09b3460965945f18bf60bbb261"}
Oct 30 02:42:03.575532 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:03.575450    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn" event={"ID":"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb","Type":"ContainerStarted","Data":"495895cd768c167eb52825da5a0731ccd66de7add9512ea57e3a767e77289d29"}
Oct 30 02:42:03.664768 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:03.664747    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg_dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8/kube-storage-version-migrator-operator/0.log"
Oct 30 02:42:04.202559 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:04.202520    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:04.257615 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:04.257572    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:42:04.257703 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:04.257626    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:04.257744 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:04.257704    4960 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:42:04.257744 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:04.257706    4960 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:42:04.257808 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:04.257749    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jwzx9\" (UID: \"70f081e1-7202-4b86-930d-8c3f72a3bb7e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:42:04.257808 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:04.257760    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls podName:6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da nodeName:}" failed. No retries permitted until 2026-10-30 02:42:08.257742759 +0000 UTC m=+8.906791159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dkfnj" (UID: "6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:42:04.257808 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:04.257793    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs podName:60924e48-a5e3-4f2b-ab35-48479c3b730b nodeName:}" failed. No retries permitted until 2026-10-30 02:42:08.257778318 +0000 UTC m=+8.906826719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs") pod "network-metrics-daemon-t5747" (UID: "60924e48-a5e3-4f2b-ab35-48479c3b730b") : secret "metrics-daemon-secret" not found
Oct 30 02:42:04.257808 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:04.257798    4960 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:42:04.257954 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:04.257827    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:42:04.257954 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:04.257840    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert podName:70f081e1-7202-4b86-930d-8c3f72a3bb7e nodeName:}" failed. No retries permitted until 2026-10-30 02:42:08.257824257 +0000 UTC m=+8.906872656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jwzx9" (UID: "70f081e1-7202-4b86-930d-8c3f72a3bb7e") : secret "networking-console-plugin-cert" not found
Oct 30 02:42:04.257954 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:04.257875    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:42:04.257954 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:04.257888    4960 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:42:04.257954 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:04.257924    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls podName:06715a40-fb77-4b9c-89e1-0aefe7b12a72 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:08.257915899 +0000 UTC m=+8.906964302 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7mfqp" (UID: "06715a40-fb77-4b9c-89e1-0aefe7b12a72") : secret "samples-operator-tls" not found
Oct 30 02:42:04.257954 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:04.257949    4960 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:42:04.258158 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:04.257970    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:42:04.258158 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:04.257980    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls podName:c5d42a36-81a2-49ad-9fa0-32d42e87d75d nodeName:}" failed. No retries permitted until 2026-10-30 02:42:08.257968021 +0000 UTC m=+8.907016432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls") pod "dns-default-7mhtz" (UID: "c5d42a36-81a2-49ad-9fa0-32d42e87d75d") : secret "dns-default-metrics-tls" not found
Oct 30 02:42:04.260749 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:04.260727    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6516df6-4df4-4213-890a-0b25e274e606-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vjvcf\" (UID: \"d6516df6-4df4-4213-890a-0b25e274e606\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:42:04.321605 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:04.321573    4960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:04.324518 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:04.324496    4960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:04.408139 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:04.408113    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:04.426170 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:04.426143    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"
Oct 30 02:42:04.517637 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:04.517610    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:04.578875 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:04.578849    4960 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:42:04.606270 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:04.606244    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:04.719069 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:04.719046    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf"]
Oct 30 02:42:04.721951 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:42:04.721928    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6516df6_4df4_4213_890a_0b25e274e606.slice/crio-777c3c012f239ce8bb2e282dc39ce26b2030e3758556b8a996cbd38ce298c920 WatchSource:0}: Error finding container 777c3c012f239ce8bb2e282dc39ce26b2030e3758556b8a996cbd38ce298c920: Status 404 returned error can't find the container with id 777c3c012f239ce8bb2e282dc39ce26b2030e3758556b8a996cbd38ce298c920
Oct 30 02:42:04.723854 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:04.723839    4960 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:42:04.804292 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:04.804221    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:05.008286 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:05.008244    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:05.423591 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:05.423557    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:05.425091 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:05.425070    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5d4bdb4f7f-89p4x"
Oct 30 02:42:05.583677 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:05.583634    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf" event={"ID":"d6516df6-4df4-4213-890a-0b25e274e606","Type":"ContainerStarted","Data":"777c3c012f239ce8bb2e282dc39ce26b2030e3758556b8a996cbd38ce298c920"}
Oct 30 02:42:05.598173 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:05.598146    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:05.696266 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:05.696201    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:05.814148 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:05.814120    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:06.202849 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:06.202826    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:06.408642 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:06.408607    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:06.588796 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:06.588756    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vjvcf" event={"ID":"d6516df6-4df4-4213-890a-0b25e274e606","Type":"ContainerStarted","Data":"9c2a980a17c2d87030844a00ef062be5a5fbaaf06a78661db26653d9461c08da"}
Oct 30 02:42:06.604602 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:06.604571    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:06.731016 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:06.730988    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:06.847399 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:06.847340    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:07.006111 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:07.006081    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:07.145347 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:07.145285    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:07.812508 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:07.812481    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:07.932405 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:07.932373    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:07.985008 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:07.984986    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:08.210139 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:08.210110    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:08.248575 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:08.248548    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:08.248913 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:08.248814    4960 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:42:08.296827 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:08.296805    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:42:08.296926 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:08.296838    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:08.296926 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:08.296861    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jwzx9\" (UID: \"70f081e1-7202-4b86-930d-8c3f72a3bb7e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:42:08.296926 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:08.296886    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:42:08.296926 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:08.296920    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:42:08.297108 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:08.296961    4960 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:42:08.297108 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:08.297015    4960 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:42:08.297108 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:08.297026    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls podName:6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da nodeName:}" failed. No retries permitted until 2026-10-30 02:42:16.297009154 +0000 UTC m=+16.946057567 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dkfnj" (UID: "6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:42:08.297108 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:08.297048    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls podName:c5d42a36-81a2-49ad-9fa0-32d42e87d75d nodeName:}" failed. No retries permitted until 2026-10-30 02:42:16.297037734 +0000 UTC m=+16.946086135 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls") pod "dns-default-7mhtz" (UID: "c5d42a36-81a2-49ad-9fa0-32d42e87d75d") : secret "dns-default-metrics-tls" not found
Oct 30 02:42:08.297108 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:08.297067    4960 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:42:08.297317 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:08.297139    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls podName:06715a40-fb77-4b9c-89e1-0aefe7b12a72 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:16.297120033 +0000 UTC m=+16.946168439 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7mfqp" (UID: "06715a40-fb77-4b9c-89e1-0aefe7b12a72") : secret "samples-operator-tls" not found
Oct 30 02:42:08.299268 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:08.299246    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70f081e1-7202-4b86-930d-8c3f72a3bb7e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jwzx9\" (UID: \"70f081e1-7202-4b86-930d-8c3f72a3bb7e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:42:08.299365 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:08.299267    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60924e48-a5e3-4f2b-ab35-48479c3b730b-metrics-certs\") pod \"network-metrics-daemon-t5747\" (UID: \"60924e48-a5e3-4f2b-ab35-48479c3b730b\") " pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:42:08.327429 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:08.327406    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"
Oct 30 02:42:08.331104 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:08.331088    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5747"
Oct 30 02:42:08.412750 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:08.412674    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:08.534523 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:08.534464    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:08.625381 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:08.625360    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t5747"]
Oct 30 02:42:08.627347 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:42:08.627322    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60924e48_a5e3_4f2b_ab35_48479c3b730b.slice/crio-32ddec4fb519781d53676afb8dd5cb47ab16ff6e78c0e91c6a551233c04e936d WatchSource:0}: Error finding container 32ddec4fb519781d53676afb8dd5cb47ab16ff6e78c0e91c6a551233c04e936d: Status 404 returned error can't find the container with id 32ddec4fb519781d53676afb8dd5cb47ab16ff6e78c0e91c6a551233c04e936d
Oct 30 02:42:08.635129 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:08.635106    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jwzx9"]
Oct 30 02:42:08.637461 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:42:08.637437    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f081e1_7202_4b86_930d_8c3f72a3bb7e.slice/crio-40d2d9c37dc3a2c0534527d7489b46d20b0477f39f0df1c00f09aa3b1c43547d WatchSource:0}: Error finding container 40d2d9c37dc3a2c0534527d7489b46d20b0477f39f0df1c00f09aa3b1c43547d: Status 404 returned error can't find the container with id 40d2d9c37dc3a2c0534527d7489b46d20b0477f39f0df1c00f09aa3b1c43547d
Oct 30 02:42:08.650684 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:08.650663    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:08.797103 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:08.797052    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:09.001991 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:09.001958    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:09.426053 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:09.426025    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fl65k"
Oct 30 02:42:09.458246 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:09.458193    4960 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:42:09.458558 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:09.458542    4960 scope.go:122] "RemoveContainer" containerID="aa43f12ca5e4cb025b1bf59fcdf00f98211ba44c2888f10f9756b2b636d7c1a2"
Oct 30 02:42:09.458758 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:09.458734    4960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-scstg_openshift-console-operator(fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg" podUID="fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e"
Oct 30 02:42:09.602482 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:09.602440    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9" event={"ID":"70f081e1-7202-4b86-930d-8c3f72a3bb7e","Type":"ContainerStarted","Data":"40d2d9c37dc3a2c0534527d7489b46d20b0477f39f0df1c00f09aa3b1c43547d"}
Oct 30 02:42:09.605278 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:09.605253    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t5747" event={"ID":"60924e48-a5e3-4f2b-ab35-48479c3b730b","Type":"ContainerStarted","Data":"8790bc065d0962ff8b2f66d43d339a6c975fea3fbad8fade25c42487b0dac3c9"}
Oct 30 02:42:09.605379 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:09.605304    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t5747" event={"ID":"60924e48-a5e3-4f2b-ab35-48479c3b730b","Type":"ContainerStarted","Data":"32ddec4fb519781d53676afb8dd5cb47ab16ff6e78c0e91c6a551233c04e936d"}
Oct 30 02:42:09.629432 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:09.629410    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:42:10.203222 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:10.203195    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:10.410078 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:10.410048    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:10.600558 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:10.600526    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:10.610426 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:10.610394    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-jwzx9" event={"ID":"70f081e1-7202-4b86-930d-8c3f72a3bb7e","Type":"ContainerStarted","Data":"6fb8f3ecdb0e37c8eb5f15818a439f9fa6522392ba4d0680052532c73c5ba63c"}
Oct 30 02:42:10.612694 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:10.612669    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t5747" event={"ID":"60924e48-a5e3-4f2b-ab35-48479c3b730b","Type":"ContainerStarted","Data":"af674e861f6d6aad82a7f9b3fc2e580b25fb65f15efb3a0f9ed8b312937b933a"}
Oct 30 02:42:10.811291 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:10.811268    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:11.017227 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:11.017158    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:11.402020 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:11.401993    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:11.496747 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:11.496726    4960 scope.go:122] "RemoveContainer" containerID="aa43f12ca5e4cb025b1bf59fcdf00f98211ba44c2888f10f9756b2b636d7c1a2"
Oct 30 02:42:11.617363 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:11.617337    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 02:42:11.617495 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:11.617452    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg" event={"ID":"fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e","Type":"ContainerStarted","Data":"8af1b8a802ff2c4b881d5f4ae5fcb51186d23740dd976b6938da036c2b211ca8"}
Oct 30 02:42:11.617937 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:11.617915    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:42:11.618752 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:11.618732    4960 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-scstg container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused" start-of-body=
Oct 30 02:42:11.618812 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:11.618776    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg" podUID="fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused"
Oct 30 02:42:12.176225 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:12.176171    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:12.301129 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:12.301100    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:12.406052 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:12.406030    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:12.515311 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:12.515253    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:12.624601 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:12.624581    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-scstg"
Oct 30 02:42:12.693077 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:12.693052    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:13.871165 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:13.871137    4960 patch_prober.go:29] interesting pod/image-registry-7c6dbd69f-lrwtw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 02:42:13.871605 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:13.871210    4960 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw" podUID="b93ba591-108f-41bd-afed-012dcd2f8690" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 02:42:14.532234 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:14.532206    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c6dbd69f-lrwtw"]
Oct 30 02:42:14.611499 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:14.611467    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:14.782598 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:14.782533    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:14.896590 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:14.896555    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:16.201916 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:16.201883    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9xpbx"
Oct 30 02:42:16.356296 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:16.356268    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:16.356447 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:16.356305    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:42:16.356447 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:16.356338    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:42:16.358843 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:16.358817    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d42a36-81a2-49ad-9fa0-32d42e87d75d-metrics-tls\") pod \"dns-default-7mhtz\" (UID: \"c5d42a36-81a2-49ad-9fa0-32d42e87d75d\") " pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:42:16.358843 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:16.358830    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkfnj\" (UID: \"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da\") " pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:16.358992 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:16.358879    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/06715a40-fb77-4b9c-89e1-0aefe7b12a72-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7mfqp\" (UID: \"06715a40-fb77-4b9c-89e1-0aefe7b12a72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:42:16.425444 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:16.425419    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"
Oct 30 02:42:16.435281 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:16.435262    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:42:16.436369 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:16.436351    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dkfnj"
Oct 30 02:42:16.728123 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:16.728104    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp"]
Oct 30 02:42:16.741985 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:16.741960    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7mhtz"]
Oct 30 02:42:16.764817 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:16.764742    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dkfnj"]
Oct 30 02:42:16.766908 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:42:16.766886    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0c92eb_fdcf_4e6d_bd2a_65a80e2872da.slice/crio-bd0ccc843be2bd47d7352ab14e8b41cadad220f938127c8dd380a1d0f904407a WatchSource:0}: Error finding container bd0ccc843be2bd47d7352ab14e8b41cadad220f938127c8dd380a1d0f904407a: Status 404 returned error can't find the container with id bd0ccc843be2bd47d7352ab14e8b41cadad220f938127c8dd380a1d0f904407a
Oct 30 02:42:17.251829 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:17.251760    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:17.379693 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:17.379624    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:17.492397 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:17.492374    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:17.638851 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:17.638776    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7mhtz" event={"ID":"c5d42a36-81a2-49ad-9fa0-32d42e87d75d","Type":"ContainerStarted","Data":"b75c061dc1768b94ba75a3afa5d0341f634ec53333a8de062286ac157d559a53"}
Oct 30 02:42:17.641058 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:17.641022    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dkfnj" event={"ID":"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da","Type":"ContainerStarted","Data":"f6dd4445bde792d7e4d760fc5dd85f306e59bbda6994eb51e87cf74e3ef50b6f"}
Oct 30 02:42:17.641167 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:17.641089    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dkfnj" event={"ID":"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da","Type":"ContainerStarted","Data":"31c56b0679e5f4df079c48b13b9e4dcddff2e6d4e2f245bbda3eced338b7aaa9"}
Oct 30 02:42:17.641167 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:17.641119    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dkfnj" event={"ID":"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da","Type":"ContainerStarted","Data":"bd0ccc843be2bd47d7352ab14e8b41cadad220f938127c8dd380a1d0f904407a"}
Oct 30 02:42:17.643071 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:17.643042    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp" event={"ID":"06715a40-fb77-4b9c-89e1-0aefe7b12a72","Type":"ContainerStarted","Data":"b4e80c0b2a2e2a3ae6c79082c0b07ff4d08f726ca6adca2fabd400f79846e8e5"}
Oct 30 02:42:18.685485 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:18.685436    4960 patch_prober.go:29] interesting pod/image-registry-59bd5b5c6b-lcbtn container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 02:42:18.685927 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:18.685494    4960 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn" podUID="bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 02:42:19.128439 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:19.128414    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:19.652998 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:19.652964    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dkfnj" event={"ID":"6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da","Type":"ContainerStarted","Data":"f901a6f247ed2d4c9bcfabf24893939f680f0dec3244744f8f7e794f4b0d0992"}
Oct 30 02:42:19.654730 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:19.654705    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp" event={"ID":"06715a40-fb77-4b9c-89e1-0aefe7b12a72","Type":"ContainerStarted","Data":"73639fbd64bd317e1944ef5b46a96ed4f438f303eb66349104a74fcd99873411"}
Oct 30 02:42:19.654855 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:19.654747    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7mfqp" event={"ID":"06715a40-fb77-4b9c-89e1-0aefe7b12a72","Type":"ContainerStarted","Data":"86c1d121f417c7b788614344ea6909ea7e50c4fa11caebf5a97ff85b5355098a"}
Oct 30 02:42:19.656408 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:19.656385    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7mhtz" event={"ID":"c5d42a36-81a2-49ad-9fa0-32d42e87d75d","Type":"ContainerStarted","Data":"b69195c13be3b6c166207d996fa90f7914eec4bc405bc5f965f5649c2b43b369"}
Oct 30 02:42:19.656473 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:19.656428    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7mhtz" event={"ID":"c5d42a36-81a2-49ad-9fa0-32d42e87d75d","Type":"ContainerStarted","Data":"afe17f3a7c37d4cc1d4101a4c42b8f408942aa030e90b8d07eb53a62b2c3d3ea"}
Oct 30 02:42:19.656473 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:19.656461    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:42:20.007452 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:20.007375    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:21.706605 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:21.706564    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:21.908904 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:21.908869    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:22.001449 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.001372    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c8988"]
Oct 30 02:42:22.004962 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.004947    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.036095 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.036066    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 02:42:22.036278 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.036256    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:22.036702 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.036682    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f69xl\""
Oct 30 02:42:22.036779 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.036732    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 02:42:22.100203 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.100157    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b94b540a-e972-41a6-8b11-e50b428e6f75-root\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.100314 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.100208    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b94b540a-e972-41a6-8b11-e50b428e6f75-node-exporter-accelerators-collector-config\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.100314 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.100298    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b94b540a-e972-41a6-8b11-e50b428e6f75-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.100382 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.100338    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b94b540a-e972-41a6-8b11-e50b428e6f75-sys\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.100382 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.100361    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b94b540a-e972-41a6-8b11-e50b428e6f75-node-exporter-textfile\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.100444 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.100391    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsnqb\" (UniqueName: \"kubernetes.io/projected/b94b540a-e972-41a6-8b11-e50b428e6f75-kube-api-access-dsnqb\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.100490 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.100477    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b94b540a-e972-41a6-8b11-e50b428e6f75-node-exporter-wtmp\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.100560 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.100545    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b94b540a-e972-41a6-8b11-e50b428e6f75-node-exporter-tls\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.100595 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.100567    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b94b540a-e972-41a6-8b11-e50b428e6f75-metrics-client-ca\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.109580 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.109562    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 02:42:22.109580 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.109565    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 02:42:22.170821 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.170795    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:22.201057 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.201026    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b94b540a-e972-41a6-8b11-e50b428e6f75-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.201148 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.201064    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b94b540a-e972-41a6-8b11-e50b428e6f75-sys\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.201148 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.201084    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b94b540a-e972-41a6-8b11-e50b428e6f75-node-exporter-textfile\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.201275 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.201156    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b94b540a-e972-41a6-8b11-e50b428e6f75-sys\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.201275 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.201214    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsnqb\" (UniqueName: \"kubernetes.io/projected/b94b540a-e972-41a6-8b11-e50b428e6f75-kube-api-access-dsnqb\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.201376 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.201279    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b94b540a-e972-41a6-8b11-e50b428e6f75-node-exporter-wtmp\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.201376 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.201341    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b94b540a-e972-41a6-8b11-e50b428e6f75-node-exporter-tls\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.201376 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.201367    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b94b540a-e972-41a6-8b11-e50b428e6f75-metrics-client-ca\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.201508 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.201420    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b94b540a-e972-41a6-8b11-e50b428e6f75-root\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.201508 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.201436    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b94b540a-e972-41a6-8b11-e50b428e6f75-node-exporter-wtmp\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.201508 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.201449    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b94b540a-e972-41a6-8b11-e50b428e6f75-node-exporter-accelerators-collector-config\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.201508 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.201440    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b94b540a-e972-41a6-8b11-e50b428e6f75-node-exporter-textfile\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.201508 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.201497    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b94b540a-e972-41a6-8b11-e50b428e6f75-root\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.201927 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.201908    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b94b540a-e972-41a6-8b11-e50b428e6f75-metrics-client-ca\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.202207 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.202166    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b94b540a-e972-41a6-8b11-e50b428e6f75-node-exporter-accelerators-collector-config\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.203653 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.203634    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b94b540a-e972-41a6-8b11-e50b428e6f75-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.203798 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.203779    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b94b540a-e972-41a6-8b11-e50b428e6f75-node-exporter-tls\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.222498 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.222472    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsnqb\" (UniqueName: \"kubernetes.io/projected/b94b540a-e972-41a6-8b11-e50b428e6f75-kube-api-access-dsnqb\") pod \"node-exporter-c8988\" (UID: \"b94b540a-e972-41a6-8b11-e50b428e6f75\") " pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.281232 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.281131    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:22.321827 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.321798    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c8988"
Oct 30 02:42:22.335331 ip-10-0-128-241 kubenswrapper[4960]: W1030 02:42:22.335301    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94b540a_e972_41a6_8b11_e50b428e6f75.slice/crio-f568cbc54e7b5728abc5cc575179fb888d1acd1227af675433aa325b7f33c9d9 WatchSource:0}: Error finding container f568cbc54e7b5728abc5cc575179fb888d1acd1227af675433aa325b7f33c9d9: Status 404 returned error can't find the container with id f568cbc54e7b5728abc5cc575179fb888d1acd1227af675433aa325b7f33c9d9
Oct 30 02:42:22.405967 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.405938    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:22.669053 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.669025    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8988" event={"ID":"b94b540a-e972-41a6-8b11-e50b428e6f75","Type":"ContainerStarted","Data":"f568cbc54e7b5728abc5cc575179fb888d1acd1227af675433aa325b7f33c9d9"}
Oct 30 02:42:22.809452 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.809414    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:22.917983 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:22.917952    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:23.174431 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:23.174392    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:23.434465 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:23.434413    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:23.622269 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:23.622242    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:23.674893 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:23.674863    4960 generic.go:356] "Generic (PLEG): container finished" podID="b94b540a-e972-41a6-8b11-e50b428e6f75" containerID="5198534f64969b4d7b37ac0666a1455799d62a02d05d1537d70914a9fab9563b" exitCode=0
Oct 30 02:42:23.675018 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:23.674899    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8988" event={"ID":"b94b540a-e972-41a6-8b11-e50b428e6f75","Type":"ContainerDied","Data":"5198534f64969b4d7b37ac0666a1455799d62a02d05d1537d70914a9fab9563b"}
Oct 30 02:42:23.953722 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:23.953638    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:42:24.537520 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:24.537492    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:24.681387 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:24.681351    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8988" event={"ID":"b94b540a-e972-41a6-8b11-e50b428e6f75","Type":"ContainerStarted","Data":"fc11006e0a81224e833aaa6dd39a7527515cae987f6c1806a5e39e90c0adc0d6"}
Oct 30 02:42:24.681533 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:24.681399    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8988" event={"ID":"b94b540a-e972-41a6-8b11-e50b428e6f75","Type":"ContainerStarted","Data":"04f86114f4c9c99a034a6a6bb19ee0c49c5e1468c74a7643fa0b5102b4981ac9"}
Oct 30 02:42:24.724103 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:24.724059    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c8988" podStartSLOduration=2.74595004 podStartE2EDuration="3.724046778s" totalImagesPullingTime="902.994915ms" totalInitContainerRuntime="75.101823ms" isStatefulPod=true podCreationTimestamp="2026-10-30 02:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:42:24.723269789 +0000 UTC m=+25.372318225" watchObservedRunningTime="2026-10-30 02:42:24.724046778 +0000 UTC m=+25.373095199"
Oct 30 02:42:25.232641 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:25.232601    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:25.337125 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:25.337095    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:25.443228 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:25.443203    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:25.587790 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:25.587770    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:42:25.863103 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:25.863028    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:25.970392 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:25.970361    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:26.793034 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:26.792998    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:26.903529 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:26.903500    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:27.093214 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:27.093174    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:28.836877 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:28.836846    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:28.977198 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:28.977145    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:29.086470 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:29.086415    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:29.632230 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:29.632205    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wx7bz"
Oct 30 02:42:29.662273 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:29.662250    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7mhtz"
Oct 30 02:42:33.421714 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:33.421676    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:33.530172 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:33.530136    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:35.164345 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:35.164309    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:35.276813 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:35.276779    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:37.356148 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:37.356115    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"]
Oct 30 02:42:37.477787 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:37.477731    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:37.576392 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:37.576358    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:37.843099 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:37.843055    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:37.958806 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:37.958778    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:39.553432 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.553390    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw" podUID="b93ba591-108f-41bd-afed-012dcd2f8690" containerName="registry" containerID="cri-o://e4e46ff6b889767ca4e25938f9a118549873e78fa46895a7dafe2d44c6a8e02d" gracePeriod=30
Oct 30 02:42:39.755051 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.755020    4960 generic.go:356] "Generic (PLEG): container finished" podID="b93ba591-108f-41bd-afed-012dcd2f8690" containerID="e4e46ff6b889767ca4e25938f9a118549873e78fa46895a7dafe2d44c6a8e02d" exitCode=0
Oct 30 02:42:39.755233 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.755070    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw" event={"ID":"b93ba591-108f-41bd-afed-012dcd2f8690","Type":"ContainerDied","Data":"e4e46ff6b889767ca4e25938f9a118549873e78fa46895a7dafe2d44c6a8e02d"}
Oct 30 02:42:39.779631 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.779609    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:39.834251 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.834227    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-installation-pull-secrets\") pod \"b93ba591-108f-41bd-afed-012dcd2f8690\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") "
Oct 30 02:42:39.834361 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.834271    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-registry-certificates\") pod \"b93ba591-108f-41bd-afed-012dcd2f8690\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") "
Oct 30 02:42:39.834361 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.834317    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93ba591-108f-41bd-afed-012dcd2f8690-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b93ba591-108f-41bd-afed-012dcd2f8690-ca-trust-extracted\") pod \"b93ba591-108f-41bd-afed-012dcd2f8690\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") "
Oct 30 02:42:39.834534 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.834505    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-trusted-ca\") pod \"b93ba591-108f-41bd-afed-012dcd2f8690\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") "
Oct 30 02:42:39.834679 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.834589    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls\") pod \"b93ba591-108f-41bd-afed-012dcd2f8690\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") "
Oct 30 02:42:39.834679 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.834620    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-bound-sa-token\") pod \"b93ba591-108f-41bd-afed-012dcd2f8690\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") "
Oct 30 02:42:39.834679 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.834641    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-kube-api-access-hmg6l\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-kube-api-access-hmg6l\") pod \"b93ba591-108f-41bd-afed-012dcd2f8690\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") "
Oct 30 02:42:39.834679 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.834673    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-image-registry-private-configuration\") pod \"b93ba591-108f-41bd-afed-012dcd2f8690\" (UID: \"b93ba591-108f-41bd-afed-012dcd2f8690\") "
Oct 30 02:42:39.834911 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.834676    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-registry-certificates" pod "b93ba591-108f-41bd-afed-012dcd2f8690" (UID: "b93ba591-108f-41bd-afed-012dcd2f8690"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:39.834911 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.834901    4960 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-registry-certificates\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 02:42:39.835037 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.835000    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-trusted-ca" pod "b93ba591-108f-41bd-afed-012dcd2f8690" (UID: "b93ba591-108f-41bd-afed-012dcd2f8690"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:39.838025 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.837947    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-bound-sa-token" pod "b93ba591-108f-41bd-afed-012dcd2f8690" (UID: "b93ba591-108f-41bd-afed-012dcd2f8690"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:42:39.838625 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.838595    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls" pod "b93ba591-108f-41bd-afed-012dcd2f8690" (UID: "b93ba591-108f-41bd-afed-012dcd2f8690"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:42:39.838984 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.838958    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-kube-api-access-hmg6l" pod "b93ba591-108f-41bd-afed-012dcd2f8690" (UID: "b93ba591-108f-41bd-afed-012dcd2f8690"). InnerVolumeSpecName "kube-api-access-hmg6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:42:39.838984 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.838972    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-installation-pull-secrets" pod "b93ba591-108f-41bd-afed-012dcd2f8690" (UID: "b93ba591-108f-41bd-afed-012dcd2f8690"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:39.839159 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.839131    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-image-registry-private-configuration" pod "b93ba591-108f-41bd-afed-012dcd2f8690" (UID: "b93ba591-108f-41bd-afed-012dcd2f8690"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:39.844583 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.844561    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93ba591-108f-41bd-afed-012dcd2f8690-ca-trust-extracted" pod "b93ba591-108f-41bd-afed-012dcd2f8690" (UID: "b93ba591-108f-41bd-afed-012dcd2f8690"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:42:39.936025 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.936003    4960 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b93ba591-108f-41bd-afed-012dcd2f8690-trusted-ca\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 02:42:39.936025 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.936023    4960 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-registry-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 02:42:39.936165 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.936032    4960 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-bound-sa-token\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 02:42:39.936165 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.936040    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmg6l\" (UniqueName: \"kubernetes.io/projected/b93ba591-108f-41bd-afed-012dcd2f8690-kube-api-access-hmg6l\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 02:42:39.936165 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.936052    4960 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-image-registry-private-configuration\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 02:42:39.936165 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.936061    4960 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b93ba591-108f-41bd-afed-012dcd2f8690-installation-pull-secrets\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 02:42:39.936165 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:39.936070    4960 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b93ba591-108f-41bd-afed-012dcd2f8690-ca-trust-extracted\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 02:42:40.442912 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:40.442850    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:40.556519 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:40.556485    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:40.760404 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:40.760323    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw" event={"ID":"b93ba591-108f-41bd-afed-012dcd2f8690","Type":"ContainerDied","Data":"f4514f7f51047f78e8144152ec61451caa464215133c5b0b41c99766df0713dd"}
Oct 30 02:42:40.760404 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:40.760343    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c6dbd69f-lrwtw"
Oct 30 02:42:40.760404 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:40.760384    4960 scope.go:122] "RemoveContainer" containerID="e4e46ff6b889767ca4e25938f9a118549873e78fa46895a7dafe2d44c6a8e02d"
Oct 30 02:42:40.827585 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:40.827560    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c6dbd69f-lrwtw"]
Oct 30 02:42:40.840969 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:40.840943    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c6dbd69f-lrwtw"]
Oct 30 02:42:41.503883 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:41.503852    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b93ba591-108f-41bd-afed-012dcd2f8690" path="/var/lib/kubelet/pods/b93ba591-108f-41bd-afed-012dcd2f8690/volumes"
Oct 30 02:42:42.268574 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:42.268541    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:42.379403 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:42.379377    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:43.622610 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:43.622573    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:43.723119 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:43.723089    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:47.673837 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:47.673789    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:47.788424 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:47.788391    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:49.412799 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:49.412769    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:49.552309 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:49.552280    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:52.526549 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:52.526512    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:52.635616 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:52.635581    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:52.791011 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:52.790952    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:52.899420 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:52.899391    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:53.953408 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:42:53.953375    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:42:57.354511 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:57.354470    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:57.472189 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:42:57.472140    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:01.806863 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:01.806824    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:01.933734 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:01.933700    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:02.376556 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.376519    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn" podUID="bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" containerName="registry" containerID="cri-o://d9856c58531e30fcc09dfd51ef493836c0303a09b3460965945f18bf60bbb261" gracePeriod=30
Oct 30 02:43:02.638798 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.638744    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:43:02.702831 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.702804    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-trusted-ca\") pod \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") "
Oct 30 02:43:02.702974 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.702886    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-installation-pull-secrets\") pod \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") "
Oct 30 02:43:02.702974 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.702909    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-kube-api-access-dxvfq\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-kube-api-access-dxvfq\") pod \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") "
Oct 30 02:43:02.702974 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.702929    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-bound-sa-token\") pod \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") "
Oct 30 02:43:02.702974 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.702960    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-certificates\") pod \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") "
Oct 30 02:43:02.703212 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.703005    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-ca-trust-extracted\") pod \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") "
Oct 30 02:43:02.703212 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.703038    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls\") pod \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") "
Oct 30 02:43:02.703212 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.703069    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-image-registry-private-configuration\") pod \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\" (UID: \"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb\") "
Oct 30 02:43:02.703411 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.703383    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-trusted-ca" pod "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" (UID: "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:43:02.704364 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.704327    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-certificates" pod "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" (UID: "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:43:02.706070 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.706026    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-installation-pull-secrets" pod "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" (UID: "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:43:02.706317 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.706280    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-image-registry-private-configuration" pod "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" (UID: "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:43:02.706465 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.706438    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-bound-sa-token" pod "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" (UID: "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:43:02.707993 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.707952    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls" pod "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" (UID: "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:43:02.708101 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.708067    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-kube-api-access-dxvfq" pod "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" (UID: "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb"). InnerVolumeSpecName "kube-api-access-dxvfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:43:02.711136 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.710982    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-ca-trust-extracted" pod "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" (UID: "bedee86a-79f7-43cf-9ed2-5dbf5b8109eb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:43:02.803878 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.803851    4960 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-installation-pull-secrets\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 02:43:02.803878 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.803872    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxvfq\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-kube-api-access-dxvfq\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 02:43:02.803878 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.803882    4960 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-bound-sa-token\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 02:43:02.804048 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.803891    4960 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-certificates\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 02:43:02.804048 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.803900    4960 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-ca-trust-extracted\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 02:43:02.804048 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.803908    4960 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-registry-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 02:43:02.804048 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.803917    4960 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-image-registry-private-configuration\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 02:43:02.804048 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.803925    4960 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb-trusted-ca\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 02:43:02.848016 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.847981    4960 generic.go:356] "Generic (PLEG): container finished" podID="bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" containerID="d9856c58531e30fcc09dfd51ef493836c0303a09b3460965945f18bf60bbb261" exitCode=0
Oct 30 02:43:02.848521 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.848079    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"
Oct 30 02:43:02.848521 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.848101    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn" event={"ID":"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb","Type":"ContainerDied","Data":"d9856c58531e30fcc09dfd51ef493836c0303a09b3460965945f18bf60bbb261"}
Oct 30 02:43:02.848521 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.848151    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59bd5b5c6b-lcbtn" event={"ID":"bedee86a-79f7-43cf-9ed2-5dbf5b8109eb","Type":"ContainerDied","Data":"495895cd768c167eb52825da5a0731ccd66de7add9512ea57e3a767e77289d29"}
Oct 30 02:43:02.848521 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.848191    4960 scope.go:122] "RemoveContainer" containerID="d9856c58531e30fcc09dfd51ef493836c0303a09b3460965945f18bf60bbb261"
Oct 30 02:43:02.857301 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.857286    4960 scope.go:122] "RemoveContainer" containerID="d9856c58531e30fcc09dfd51ef493836c0303a09b3460965945f18bf60bbb261"
Oct 30 02:43:02.857552 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:43:02.857534    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9856c58531e30fcc09dfd51ef493836c0303a09b3460965945f18bf60bbb261\": container with ID starting with d9856c58531e30fcc09dfd51ef493836c0303a09b3460965945f18bf60bbb261 not found: ID does not exist" containerID="d9856c58531e30fcc09dfd51ef493836c0303a09b3460965945f18bf60bbb261"
Oct 30 02:43:02.857596 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.857562    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9856c58531e30fcc09dfd51ef493836c0303a09b3460965945f18bf60bbb261"} err="failed to get container status \"d9856c58531e30fcc09dfd51ef493836c0303a09b3460965945f18bf60bbb261\": rpc error: code = NotFound desc = could not find container \"d9856c58531e30fcc09dfd51ef493836c0303a09b3460965945f18bf60bbb261\": container with ID starting with d9856c58531e30fcc09dfd51ef493836c0303a09b3460965945f18bf60bbb261 not found: ID does not exist"
Oct 30 02:43:02.900669 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.900619    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"]
Oct 30 02:43:02.904578 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:02.904560    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59bd5b5c6b-lcbtn"]
Oct 30 02:43:03.501620 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:03.501579    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" path="/var/lib/kubelet/pods/bedee86a-79f7-43cf-9ed2-5dbf5b8109eb/volumes"
Oct 30 02:43:07.834425 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:07.834390    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:07.935076 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:07.935041    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:08.205073 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:08.204990    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:08.304354 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:08.304319    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:09.764461 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:09.764425    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:14.899963 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:14.899929    4960 generic.go:356] "Generic (PLEG): container finished" podID="dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8" containerID="40f41f00c8f15be84f03f06c759fd7ae3540edc20469164b67bf278dba9a2e0e" exitCode=0
Oct 30 02:43:14.900358 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:14.899981    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg" event={"ID":"dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8","Type":"ContainerDied","Data":"40f41f00c8f15be84f03f06c759fd7ae3540edc20469164b67bf278dba9a2e0e"}
Oct 30 02:43:14.900358 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:14.900338    4960 scope.go:122] "RemoveContainer" containerID="40f41f00c8f15be84f03f06c759fd7ae3540edc20469164b67bf278dba9a2e0e"
Oct 30 02:43:15.906618 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:15.906581    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-4gtrg" event={"ID":"dd14a05e-69c0-4a9a-99d8-0ff750ceb3c8","Type":"ContainerStarted","Data":"5e0edd08271a8b57b0c31f0cd2803e3846a92c0dc3beb7afca414f148294a287"}
Oct 30 02:43:22.799612 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:22.799578    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:22.913150 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:22.913121    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:23.952837 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:43:23.952803    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:43:24.607972 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:24.607918    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:24.710564 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:24.710536    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:26.449461 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:26.449432    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:28.955071 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:28.955041    4960 generic.go:356] "Generic (PLEG): container finished" podID="f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9" containerID="a3a54794c55b68f6001cc496d8d6f6debe5488ea5b0320406d11685bf112882b" exitCode=0
Oct 30 02:43:28.955443 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:28.955118    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-gxb72" event={"ID":"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9","Type":"ContainerDied","Data":"a3a54794c55b68f6001cc496d8d6f6debe5488ea5b0320406d11685bf112882b"}
Oct 30 02:43:28.955443 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:28.955406    4960 scope.go:122] "RemoveContainer" containerID="a3a54794c55b68f6001cc496d8d6f6debe5488ea5b0320406d11685bf112882b"
Oct 30 02:43:29.960771 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:29.960735    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-gxb72" event={"ID":"f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9","Type":"ContainerStarted","Data":"bf5b4ba1801aa4109265f925c5abe10a5402d0c02a252a735463f00ea0ca6848"}
Oct 30 02:43:29.962154 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:29.962128    4960 generic.go:356] "Generic (PLEG): container finished" podID="88d453b9-7e71-4561-8e1b-4b4074651675" containerID="942369d20c6232a75314f6e6ebe1d5787995fb081871a51c5ec041bc2e7cae50" exitCode=0
Oct 30 02:43:29.962282 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:29.962171    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s" event={"ID":"88d453b9-7e71-4561-8e1b-4b4074651675","Type":"ContainerDied","Data":"942369d20c6232a75314f6e6ebe1d5787995fb081871a51c5ec041bc2e7cae50"}
Oct 30 02:43:29.962458 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:29.962444    4960 scope.go:122] "RemoveContainer" containerID="942369d20c6232a75314f6e6ebe1d5787995fb081871a51c5ec041bc2e7cae50"
Oct 30 02:43:30.968686 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:30.968643    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vbz2s" event={"ID":"88d453b9-7e71-4561-8e1b-4b4074651675","Type":"ContainerStarted","Data":"b9028e904b0f1c5e05114fc59228915939c473cf1c78ab4f9cae973993d8ad84"}
Oct 30 02:43:33.261486 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:33.261440    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:33.378737 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:33.378704    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:33.485348 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:33.485317    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:33.706570 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:33.706545    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:33.807101 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:33.807065    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:45.790902 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:45.790857    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:45.897715 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:45.897686    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:49.520828 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:49.520800    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:52.790899 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:52.790818    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:52.891474 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:52.891446    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:53.011325 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:43:53.011301    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:53.953953 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:43:53.953917    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:44:11.938359 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:44:11.938272    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:44:23.953282 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:44:23.953235    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:44:49.653293 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:44:49.653244    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:44:53.952991 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:44:53.952952    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:44:57.310114 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:44:57.310081    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:44:57.424231 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:44:57.424172    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:23.953227 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:45:23.953173    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:45:29.886875 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:45:29.886787    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:30.000145 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:45:30.000110    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:49.760936 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:45:49.760889    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:53.952704 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:45:53.952667    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:46:12.496743 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:46:12.496704    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:12.616636 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:46:12.616600    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:23.954310 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:46:23.954268    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:46:49.878224 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:46:49.878166    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:53.952352 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:46:53.952311    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:46:59.519716 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:46:59.519688    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 02:46:59.520144 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:46:59.519720    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 02:46:59.525849 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:46:59.525830    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 02:46:59.525849 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:46:59.525848    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 02:47:23.952311 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:47:23.952236    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:47:46.914262 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:47:46.914220    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:47:47.039040 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:47:47.039011    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:47:49.983765 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:47:49.983730    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:47:53.952768 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:47:53.952731    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:48:00.825374 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:48:00.825341    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:00.937251 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:48:00.937169    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:23.954924 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:48:23.954878    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:48:50.113139 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:48:50.113105    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:50.241017 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:48:50.240968    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:53.951979 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:48:53.951945    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:49:23.951780 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:49:23.951744    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:49:50.238926 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:49:50.238891    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:53.953649 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:49:53.953604    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:50:23.953330 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:50:23.953252    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:50:50.342667 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:50:50.342634    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:50:53.954409 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:50:53.954366    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:51:23.956310 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:51:23.956171    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:51:50.451418 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:51:50.451373    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:51:53.953385 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:51:53.953352    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:51:59.542440 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:51:59.542413    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 02:51:59.543355 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:51:59.543332    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 02:51:59.548236 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:51:59.548216    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 02:51:59.549101 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:51:59.549078    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 02:52:23.952413 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:52:23.952375    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:52:50.652876 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:52:50.652835    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:52:53.953220 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:52:53.953158    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:53:09.835809 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:53:09.835762    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:16.968966 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:53:16.968891    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:23.953028 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:53:23.952991    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:53:50.717857 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:53:50.717826    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:53.951978 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:53:53.951936    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:53:58.492554 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:53:58.492515    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:23.953126 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:54:23.953087    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:54:50.818655 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:54:50.818625    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:53.952714 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:54:53.952675    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:55:23.953118 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:55:23.953081    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:55:50.927072 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:55:50.927036    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:53.952701 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:55:53.952664    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:56:23.956675 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:56:23.956554    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:56:51.059901 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:56:51.059857    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:53.951965 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:56:53.951931    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:56:59.565207 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:56:59.565160    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 02:56:59.567212 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:56:59.567168    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 02:56:59.571219 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:56:59.571198    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 02:56:59.573270 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:56:59.573252    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 02:57:23.952212 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:57:23.952161    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:57:51.209101 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:57:51.209060    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:53.952711 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:57:53.952677    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:58:23.952400 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:58:23.952363    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:58:51.290263 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:58:51.290188    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:53.952501 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:58:53.952468    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:59:23.952647 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:59:23.952565    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 02:59:51.384096 ip-10-0-128-241 kubenswrapper[4960]: I1030 02:59:51.384064    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:53.952573 ip-10-0-128-241 kubenswrapper[4960]: E1030 02:59:53.952534    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:00:15.238560 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:00:15.238514    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:19.894747 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:00:19.894707    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:23.952282 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:00:23.952245    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:00:51.556853 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:00:51.556818    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:53.953649 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:00:53.953615    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:01:23.953298 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:01:23.953253    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:01:51.652653 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:01:51.652620    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:53.951465 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:01:53.951424    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:01:59.586900 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:01:59.586870    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:01:59.591258 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:01:59.591235    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:01:59.593138 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:01:59.593113    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:01:59.597251 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:01:59.597233    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:02:23.955640 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:02:23.955558    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:02:51.783915 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:02:51.783880    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:53.953246 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:02:53.953209    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:03:23.952331 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:03:23.952293    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:03:51.922741 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:03:51.922696    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:53.954593 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:03:53.954550    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:04:23.956803 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:04:23.956764    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:04:53.122065 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:04:53.122027    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:53.953062 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:04:53.953024    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:05:23.952088 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:05:23.952011    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:05:35.551754 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:05:35.551704    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:47.248905 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:05:47.248862    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:52.118275 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:05:52.118241    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:53.953501 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:05:53.953467    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:06:12.832421 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:06:12.832382    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:23.953076 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:06:23.953043    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:06:52.228744 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:06:52.228709    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:53.953054 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:06:53.953022    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:06:59.609780 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:06:59.609752    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:06:59.615358 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:06:59.615328    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:06:59.616434 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:06:59.616418    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:06:59.621786 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:06:59.621765    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:07:23.955654 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:07:23.955613    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:07:52.473692 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:07:52.473607    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:53.953974 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:07:53.953941    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:08:23.952570 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:08:23.952494    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:08:37.989406 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:08:37.989357    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:38.153191 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:08:38.153156    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:52.584072 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:08:52.584037    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:53.954768 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:08:53.954727    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:09:00.275287 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:00.275250    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:01.095079 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:01.095032    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:03.512549 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:03.512502    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:03.628082 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:03.628050    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:06.206103 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:06.206055    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:06.395417 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:06.395339    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:08.427491 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:08.427423    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:08.782127 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:08.782020    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:14.294005 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.293967    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-84gjw"]
Oct 30 03:09:14.294527 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.294286    4960 cpu_manager.go:414] "containerMap: removing container" podUID="bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" containerName="registry"
Oct 30 03:09:14.294527 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.294298    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" containerName="registry"
Oct 30 03:09:14.294527 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.294312    4960 cpu_manager.go:414] "containerMap: removing container" podUID="b93ba591-108f-41bd-afed-012dcd2f8690" containerName="registry"
Oct 30 03:09:14.294527 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.294318    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b93ba591-108f-41bd-afed-012dcd2f8690" containerName="registry"
Oct 30 03:09:14.294527 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.294369    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="bedee86a-79f7-43cf-9ed2-5dbf5b8109eb" containerName="registry"
Oct 30 03:09:14.294527 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.294378    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="b93ba591-108f-41bd-afed-012dcd2f8690" containerName="registry"
Oct 30 03:09:14.297148 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.297128    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-84gjw"
Oct 30 03:09:14.334192 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.334147    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-2qm9x\""
Oct 30 03:09:14.334308 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.334265    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 03:09:14.334308 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.334278    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 03:09:14.335074 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.335053    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-84gjw"]
Oct 30 03:09:14.464873 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.464841    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d975a4e3-008d-4d9e-889b-8d1f14f675c6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-84gjw\" (UID: \"d975a4e3-008d-4d9e-889b-8d1f14f675c6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-84gjw"
Oct 30 03:09:14.464997 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.464935    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj88h\" (UniqueName: \"kubernetes.io/projected/d975a4e3-008d-4d9e-889b-8d1f14f675c6-kube-api-access-sj88h\") pod \"cert-manager-webhook-5dbbbf75cc-84gjw\" (UID: \"d975a4e3-008d-4d9e-889b-8d1f14f675c6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-84gjw"
Oct 30 03:09:14.566169 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.566141    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj88h\" (UniqueName: \"kubernetes.io/projected/d975a4e3-008d-4d9e-889b-8d1f14f675c6-kube-api-access-sj88h\") pod \"cert-manager-webhook-5dbbbf75cc-84gjw\" (UID: \"d975a4e3-008d-4d9e-889b-8d1f14f675c6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-84gjw"
Oct 30 03:09:14.566288 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.566188    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d975a4e3-008d-4d9e-889b-8d1f14f675c6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-84gjw\" (UID: \"d975a4e3-008d-4d9e-889b-8d1f14f675c6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-84gjw"
Oct 30 03:09:14.588862 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.588834    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj88h\" (UniqueName: \"kubernetes.io/projected/d975a4e3-008d-4d9e-889b-8d1f14f675c6-kube-api-access-sj88h\") pod \"cert-manager-webhook-5dbbbf75cc-84gjw\" (UID: \"d975a4e3-008d-4d9e-889b-8d1f14f675c6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-84gjw"
Oct 30 03:09:14.590193 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.590148    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d975a4e3-008d-4d9e-889b-8d1f14f675c6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-84gjw\" (UID: \"d975a4e3-008d-4d9e-889b-8d1f14f675c6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-84gjw"
Oct 30 03:09:14.620096 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.620073    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-84gjw"
Oct 30 03:09:14.911725 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.911694    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-84gjw"]
Oct 30 03:09:14.913168 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:09:14.913141    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd975a4e3_008d_4d9e_889b_8d1f14f675c6.slice/crio-451485d1229b27b0328b40e7d837ed8b97a81bd9dbbae428be6617151c42b50f WatchSource:0}: Error finding container 451485d1229b27b0328b40e7d837ed8b97a81bd9dbbae428be6617151c42b50f: Status 404 returned error can't find the container with id 451485d1229b27b0328b40e7d837ed8b97a81bd9dbbae428be6617151c42b50f
Oct 30 03:09:14.915145 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:14.915129    4960 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:09:15.365921 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:15.365885    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-84gjw" event={"ID":"d975a4e3-008d-4d9e-889b-8d1f14f675c6","Type":"ContainerStarted","Data":"451485d1229b27b0328b40e7d837ed8b97a81bd9dbbae428be6617151c42b50f"}
Oct 30 03:09:16.890922 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:16.890883    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:17.995965 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:17.995843    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:18.124377 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:18.124339    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:18.381170 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:18.381143    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-84gjw" event={"ID":"d975a4e3-008d-4d9e-889b-8d1f14f675c6","Type":"ContainerStarted","Data":"ba36d16baa0078bd84e01505564a626eafd9e8ea3d9dc01b8346403488233cff"}
Oct 30 03:09:18.381325 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:18.381273    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-84gjw"
Oct 30 03:09:18.423482 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:18.423442    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-84gjw" podStartSLOduration=1.682331584 podStartE2EDuration="4.423430031s" totalImagesPullingTime="2.741098447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:18.42281142 +0000 UTC m=+1639.071859841" watchObservedRunningTime="2026-10-30 03:09:18.423430031 +0000 UTC m=+1639.072478451"
Oct 30 03:09:23.952055 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:09:23.952014    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:09:24.386861 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:24.386832    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-84gjw"
Oct 30 03:09:28.729476 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:28.729417    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:28.793591 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:28.793558    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-observability-system/odh-observability-5db8f88448-5w2qq"]
Oct 30 03:09:28.800642 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:28.800612    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-observability-system/odh-observability-5db8f88448-5w2qq"
Oct 30 03:09:28.820719 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:28.820696    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-observability-system/odh-observability-5db8f88448-5w2qq"]
Oct 30 03:09:28.832822 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:28.832797    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-observability-system\"/\"odh-observability-webhook-cert\""
Oct 30 03:09:28.832997 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:28.832981    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-observability-system\"/\"odh-observability-dockercfg-8lcw9\""
Oct 30 03:09:28.833050 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:28.833041    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-observability-system\"/\"odh-observability-metrics-tls\""
Oct 30 03:09:28.833160 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:28.833147    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-observability-system\"/\"kube-root-ca.crt\""
Oct 30 03:09:28.833263 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:28.833249    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-observability-system\"/\"openshift-service-ca.crt\""
Oct 30 03:09:28.878675 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:28.878652    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:28.968054 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:28.968031    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-metrics-tls\") pod \"odh-observability-5db8f88448-5w2qq\" (UID: \"62e56e71-7569-404d-a541-3d8520aea56c\") " pod="odh-observability-system/odh-observability-5db8f88448-5w2qq"
Oct 30 03:09:28.968149 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:28.968061    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-webhook-certs\") pod \"odh-observability-5db8f88448-5w2qq\" (UID: \"62e56e71-7569-404d-a541-3d8520aea56c\") " pod="odh-observability-system/odh-observability-5db8f88448-5w2qq"
Oct 30 03:09:28.968149 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:28.968088    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt2t8\" (UniqueName: \"kubernetes.io/projected/62e56e71-7569-404d-a541-3d8520aea56c-kube-api-access-nt2t8\") pod \"odh-observability-5db8f88448-5w2qq\" (UID: \"62e56e71-7569-404d-a541-3d8520aea56c\") " pod="odh-observability-system/odh-observability-5db8f88448-5w2qq"
Oct 30 03:09:29.001105 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:29.001056    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:29.068627 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:29.068600    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt2t8\" (UniqueName: \"kubernetes.io/projected/62e56e71-7569-404d-a541-3d8520aea56c-kube-api-access-nt2t8\") pod \"odh-observability-5db8f88448-5w2qq\" (UID: \"62e56e71-7569-404d-a541-3d8520aea56c\") " pod="odh-observability-system/odh-observability-5db8f88448-5w2qq"
Oct 30 03:09:29.068718 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:29.068651    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-metrics-tls\") pod \"odh-observability-5db8f88448-5w2qq\" (UID: \"62e56e71-7569-404d-a541-3d8520aea56c\") " pod="odh-observability-system/odh-observability-5db8f88448-5w2qq"
Oct 30 03:09:29.068718 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:29.068671    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-webhook-certs\") pod \"odh-observability-5db8f88448-5w2qq\" (UID: \"62e56e71-7569-404d-a541-3d8520aea56c\") " pod="odh-observability-system/odh-observability-5db8f88448-5w2qq"
Oct 30 03:09:29.068789 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:09:29.068754    4960 secret.go:189] Couldn't get secret odh-observability-system/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found
Oct 30 03:09:29.068826 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:09:29.068799    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-webhook-certs podName:62e56e71-7569-404d-a541-3d8520aea56c nodeName:}" failed. No retries permitted until 2026-10-30 03:09:29.568786217 +0000 UTC m=+1650.217834617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-webhook-certs") pod "odh-observability-5db8f88448-5w2qq" (UID: "62e56e71-7569-404d-a541-3d8520aea56c") : secret "odh-observability-webhook-cert" not found
Oct 30 03:09:29.071240 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:29.071218    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-metrics-tls\") pod \"odh-observability-5db8f88448-5w2qq\" (UID: \"62e56e71-7569-404d-a541-3d8520aea56c\") " pod="odh-observability-system/odh-observability-5db8f88448-5w2qq"
Oct 30 03:09:29.091914 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:29.091890    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt2t8\" (UniqueName: \"kubernetes.io/projected/62e56e71-7569-404d-a541-3d8520aea56c-kube-api-access-nt2t8\") pod \"odh-observability-5db8f88448-5w2qq\" (UID: \"62e56e71-7569-404d-a541-3d8520aea56c\") " pod="odh-observability-system/odh-observability-5db8f88448-5w2qq"
Oct 30 03:09:29.571510 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:29.571486    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-webhook-certs\") pod \"odh-observability-5db8f88448-5w2qq\" (UID: \"62e56e71-7569-404d-a541-3d8520aea56c\") " pod="odh-observability-system/odh-observability-5db8f88448-5w2qq"
Oct 30 03:09:29.571626 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:09:29.571575    4960 secret.go:189] Couldn't get secret odh-observability-system/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found
Oct 30 03:09:29.571626 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:09:29.571620    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-webhook-certs podName:62e56e71-7569-404d-a541-3d8520aea56c nodeName:}" failed. No retries permitted until 2026-10-30 03:09:30.571607453 +0000 UTC m=+1651.220655852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-webhook-certs") pod "odh-observability-5db8f88448-5w2qq" (UID: "62e56e71-7569-404d-a541-3d8520aea56c") : secret "odh-observability-webhook-cert" not found
Oct 30 03:09:30.577874 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:30.577837    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-webhook-certs\") pod \"odh-observability-5db8f88448-5w2qq\" (UID: \"62e56e71-7569-404d-a541-3d8520aea56c\") " pod="odh-observability-system/odh-observability-5db8f88448-5w2qq"
Oct 30 03:09:30.578304 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:09:30.577948    4960 secret.go:189] Couldn't get secret odh-observability-system/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found
Oct 30 03:09:30.578304 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:09:30.578016    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-webhook-certs podName:62e56e71-7569-404d-a541-3d8520aea56c nodeName:}" failed. No retries permitted until 2026-10-30 03:09:32.577998995 +0000 UTC m=+1653.227047394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-webhook-certs") pod "odh-observability-5db8f88448-5w2qq" (UID: "62e56e71-7569-404d-a541-3d8520aea56c") : secret "odh-observability-webhook-cert" not found
Oct 30 03:09:32.590338 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:32.590304    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-webhook-certs\") pod \"odh-observability-5db8f88448-5w2qq\" (UID: \"62e56e71-7569-404d-a541-3d8520aea56c\") " pod="odh-observability-system/odh-observability-5db8f88448-5w2qq"
Oct 30 03:09:32.590727 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:09:32.590439    4960 secret.go:189] Couldn't get secret odh-observability-system/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found
Oct 30 03:09:32.590727 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:09:32.590496    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-webhook-certs podName:62e56e71-7569-404d-a541-3d8520aea56c nodeName:}" failed. No retries permitted until 2026-10-30 03:09:36.590480268 +0000 UTC m=+1657.239528680 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-webhook-certs") pod "odh-observability-5db8f88448-5w2qq" (UID: "62e56e71-7569-404d-a541-3d8520aea56c") : secret "odh-observability-webhook-cert" not found
Oct 30 03:09:36.618096 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:36.618062    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-webhook-certs\") pod \"odh-observability-5db8f88448-5w2qq\" (UID: \"62e56e71-7569-404d-a541-3d8520aea56c\") " pod="odh-observability-system/odh-observability-5db8f88448-5w2qq"
Oct 30 03:09:36.618559 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:09:36.618200    4960 secret.go:189] Couldn't get secret odh-observability-system/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found
Oct 30 03:09:36.618559 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:09:36.618257    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-webhook-certs podName:62e56e71-7569-404d-a541-3d8520aea56c nodeName:}" failed. No retries permitted until 2026-10-30 03:09:44.618244518 +0000 UTC m=+1665.267292917 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-webhook-certs") pod "odh-observability-5db8f88448-5w2qq" (UID: "62e56e71-7569-404d-a541-3d8520aea56c") : secret "odh-observability-webhook-cert" not found
Oct 30 03:09:41.371141 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:41.371063    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:44.676573 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:44.676540    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-webhook-certs\") pod \"odh-observability-5db8f88448-5w2qq\" (UID: \"62e56e71-7569-404d-a541-3d8520aea56c\") " pod="odh-observability-system/odh-observability-5db8f88448-5w2qq"
Oct 30 03:09:44.679019 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:44.678999    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/62e56e71-7569-404d-a541-3d8520aea56c-webhook-certs\") pod \"odh-observability-5db8f88448-5w2qq\" (UID: \"62e56e71-7569-404d-a541-3d8520aea56c\") " pod="odh-observability-system/odh-observability-5db8f88448-5w2qq"
Oct 30 03:09:44.712856 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:44.712835    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-observability-system/odh-observability-5db8f88448-5w2qq"
Oct 30 03:09:45.005617 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:45.005573    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-observability-system/odh-observability-5db8f88448-5w2qq"]
Oct 30 03:09:45.007726 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:09:45.007697    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e56e71_7569_404d_a541_3d8520aea56c.slice/crio-04184f25001c13b015875ab2923c25475a62561e85b29ece04b576739d60fc4b WatchSource:0}: Error finding container 04184f25001c13b015875ab2923c25475a62561e85b29ece04b576739d60fc4b: Status 404 returned error can't find the container with id 04184f25001c13b015875ab2923c25475a62561e85b29ece04b576739d60fc4b
Oct 30 03:09:45.475870 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:45.475832    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-observability-system/odh-observability-5db8f88448-5w2qq" event={"ID":"62e56e71-7569-404d-a541-3d8520aea56c","Type":"ContainerStarted","Data":"04184f25001c13b015875ab2923c25475a62561e85b29ece04b576739d60fc4b"}
Oct 30 03:09:48.490631 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:48.490597    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-observability-system/odh-observability-5db8f88448-5w2qq" event={"ID":"62e56e71-7569-404d-a541-3d8520aea56c","Type":"ContainerStarted","Data":"bdb3db43f957ad12c087d66670b373ea5018ba337652ef2b1d9ac12517b482ae"}
Oct 30 03:09:48.491099 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:48.490733    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-observability-system/odh-observability-5db8f88448-5w2qq"
Oct 30 03:09:48.536001 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:48.535944    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-observability-system/odh-observability-5db8f88448-5w2qq" podStartSLOduration=18.033248311 podStartE2EDuration="20.535926399s" totalImagesPullingTime="2.502678088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:48.534442468 +0000 UTC m=+1669.183490889" watchObservedRunningTime="2026-10-30 03:09:48.535926399 +0000 UTC m=+1669.184974821"
Oct 30 03:09:52.619346 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:52.619309    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:53.952594 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:09:53.952552    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:09:59.501330 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:09:59.501304    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-observability-system/odh-observability-5db8f88448-5w2qq"
Oct 30 03:10:13.228023 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:10:13.227991    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:23.952531 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:10:23.952495    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:10:52.693368 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:10:52.693332    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:53.953011 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:10:53.952971    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:11:08.806814 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:08.806775    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:08.951941 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:08.951903    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:15.950344 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:15.950243    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:16.094864 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:16.094828    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:16.801280 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:16.801248    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:16.952791 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:16.952752    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:17.621963 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:17.621932    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:18.068833 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:18.068798    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:18.228515 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:18.228479    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:20.015602 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:20.015571    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:20.676735 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:20.676675    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:20.847104 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:20.847059    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:20.860475 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:20.860447    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-4qbkh"]
Oct 30 03:11:20.867410 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:20.864729    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-4qbkh"
Oct 30 03:11:20.889897 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:20.889875    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:11:20.890011 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:20.889912    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"kube-root-ca.crt\""
Oct 30 03:11:20.903924 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:20.903896    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-4qbkh"]
Oct 30 03:11:20.953243 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:20.953168    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-webhook-service-cert\""
Oct 30 03:11:20.953243 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:20.953195    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-controller-manager-dockercfg-j5t2z\""
Oct 30 03:11:21.012165 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:21.012141    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:21.037662 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:21.037636    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac86217e-b7c2-47ee-8f34-e67a4017d288-webhook-cert\") pod \"opentelemetry-operator-webhook-55c856bfb8-4qbkh\" (UID: \"ac86217e-b7c2-47ee-8f34-e67a4017d288\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-4qbkh"
Oct 30 03:11:21.037966 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:21.037672    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4rqf\" (UniqueName: \"kubernetes.io/projected/ac86217e-b7c2-47ee-8f34-e67a4017d288-kube-api-access-z4rqf\") pod \"opentelemetry-operator-webhook-55c856bfb8-4qbkh\" (UID: \"ac86217e-b7c2-47ee-8f34-e67a4017d288\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-4qbkh"
Oct 30 03:11:21.037966 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:21.037696    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ac86217e-b7c2-47ee-8f34-e67a4017d288-apiservice-cert\") pod \"opentelemetry-operator-webhook-55c856bfb8-4qbkh\" (UID: \"ac86217e-b7c2-47ee-8f34-e67a4017d288\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-4qbkh"
Oct 30 03:11:21.138944 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:21.138915    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ac86217e-b7c2-47ee-8f34-e67a4017d288-apiservice-cert\") pod \"opentelemetry-operator-webhook-55c856bfb8-4qbkh\" (UID: \"ac86217e-b7c2-47ee-8f34-e67a4017d288\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-4qbkh"
Oct 30 03:11:21.139039 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:21.138977    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac86217e-b7c2-47ee-8f34-e67a4017d288-webhook-cert\") pod \"opentelemetry-operator-webhook-55c856bfb8-4qbkh\" (UID: \"ac86217e-b7c2-47ee-8f34-e67a4017d288\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-4qbkh"
Oct 30 03:11:21.139039 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:21.139023    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4rqf\" (UniqueName: \"kubernetes.io/projected/ac86217e-b7c2-47ee-8f34-e67a4017d288-kube-api-access-z4rqf\") pod \"opentelemetry-operator-webhook-55c856bfb8-4qbkh\" (UID: \"ac86217e-b7c2-47ee-8f34-e67a4017d288\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-4qbkh"
Oct 30 03:11:21.142085 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:21.142059    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ac86217e-b7c2-47ee-8f34-e67a4017d288-apiservice-cert\") pod \"opentelemetry-operator-webhook-55c856bfb8-4qbkh\" (UID: \"ac86217e-b7c2-47ee-8f34-e67a4017d288\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-4qbkh"
Oct 30 03:11:21.142199 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:21.142163    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac86217e-b7c2-47ee-8f34-e67a4017d288-webhook-cert\") pod \"opentelemetry-operator-webhook-55c856bfb8-4qbkh\" (UID: \"ac86217e-b7c2-47ee-8f34-e67a4017d288\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-4qbkh"
Oct 30 03:11:21.163772 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:21.163742    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4rqf\" (UniqueName: \"kubernetes.io/projected/ac86217e-b7c2-47ee-8f34-e67a4017d288-kube-api-access-z4rqf\") pod \"opentelemetry-operator-webhook-55c856bfb8-4qbkh\" (UID: \"ac86217e-b7c2-47ee-8f34-e67a4017d288\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-4qbkh"
Oct 30 03:11:21.178508 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:21.178479    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-4qbkh"
Oct 30 03:11:21.500387 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:11:21.500358    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac86217e_b7c2_47ee_8f34_e67a4017d288.slice/crio-926bb99633d92b3a4c101bf21263a77c7611d9617b7005fa6992ff1691b9733f WatchSource:0}: Error finding container 926bb99633d92b3a4c101bf21263a77c7611d9617b7005fa6992ff1691b9733f: Status 404 returned error can't find the container with id 926bb99633d92b3a4c101bf21263a77c7611d9617b7005fa6992ff1691b9733f
Oct 30 03:11:21.502397 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:21.502364    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-4qbkh"]
Oct 30 03:11:21.847972 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:21.847941    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-4qbkh" event={"ID":"ac86217e-b7c2-47ee-8f34-e67a4017d288","Type":"ContainerStarted","Data":"926bb99633d92b3a4c101bf21263a77c7611d9617b7005fa6992ff1691b9733f"}
Oct 30 03:11:22.423450 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:22.423409    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:22.596236 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:22.596167    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:23.754736 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:23.754688    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:23.857499 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:23.857464    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-4qbkh" event={"ID":"ac86217e-b7c2-47ee-8f34-e67a4017d288","Type":"ContainerStarted","Data":"ba235a71a9086616b645a31a1095139a00cd17255dc5f853b2f3e02a8ba2b740"}
Oct 30 03:11:23.857667 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:23.857557    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-4qbkh"
Oct 30 03:11:23.954978 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:11:23.954945    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:11:24.003086 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:24.003057    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:26.965391 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:26.965355    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:27.395292 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:27.395257    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:27.582461 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:27.582432    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:27.941475 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:27.941444    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:29.043258 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:29.043223    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:29.237473 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:29.237439    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:30.073913 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:30.073874    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:30.290591 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:30.290549    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:30.967783 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:30.967751    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:31.122423 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:31.122392    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:31.436944 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:31.436911    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:31.850316 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:31.850285    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:32.030991 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:32.030963    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:33.463270 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:33.463235    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:33.741327 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:33.741239    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:33.981433 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:33.981399    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:34.193533 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:34.193500    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:34.354990 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:34.354960    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:34.516311 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:34.516247    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:34.682356 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:34.682324    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:34.865170 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:34.865137    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-4qbkh"
Oct 30 03:11:34.930421 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:34.930376    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-4qbkh" podStartSLOduration=13.566629348 podStartE2EDuration="14.930364291s" totalImagesPullingTime="1.363734943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:23.943056147 +0000 UTC m=+1764.592104567" watchObservedRunningTime="2026-10-30 03:11:34.930364291 +0000 UTC m=+1775.579412706"
Oct 30 03:11:35.054379 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:35.054353    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr"]
Oct 30 03:11:35.057679 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:35.057662    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr"
Oct 30 03:11:35.082779 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:35.082726    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:35.083333 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:35.083314    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr"]
Oct 30 03:11:35.147357 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:35.147298    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-admission-webhook-dockercfg-lrnv7\""
Oct 30 03:11:35.147357 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:35.147352    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-admission-webhook-service-cert\""
Oct 30 03:11:35.242690 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:35.242655    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46a4cb6f-f62d-4c7e-b4d4-3aaeb1e2c3de-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr\" (UID: \"46a4cb6f-f62d-4c7e-b4d4-3aaeb1e2c3de\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr"
Oct 30 03:11:35.242822 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:35.242713    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/46a4cb6f-f62d-4c7e-b4d4-3aaeb1e2c3de-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr\" (UID: \"46a4cb6f-f62d-4c7e-b4d4-3aaeb1e2c3de\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr"
Oct 30 03:11:35.344100 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:35.344071    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46a4cb6f-f62d-4c7e-b4d4-3aaeb1e2c3de-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr\" (UID: \"46a4cb6f-f62d-4c7e-b4d4-3aaeb1e2c3de\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr"
Oct 30 03:11:35.344236 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:35.344125    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/46a4cb6f-f62d-4c7e-b4d4-3aaeb1e2c3de-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr\" (UID: \"46a4cb6f-f62d-4c7e-b4d4-3aaeb1e2c3de\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr"
Oct 30 03:11:35.346510 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:35.346488    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/46a4cb6f-f62d-4c7e-b4d4-3aaeb1e2c3de-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr\" (UID: \"46a4cb6f-f62d-4c7e-b4d4-3aaeb1e2c3de\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr"
Oct 30 03:11:35.346588 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:35.346523    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46a4cb6f-f62d-4c7e-b4d4-3aaeb1e2c3de-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr\" (UID: \"46a4cb6f-f62d-4c7e-b4d4-3aaeb1e2c3de\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr"
Oct 30 03:11:35.369580 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:35.369561    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr"
Oct 30 03:11:35.698535 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:35.698516    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr"]
Oct 30 03:11:35.700161 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:11:35.700133    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a4cb6f_f62d_4c7e_b4d4_3aaeb1e2c3de.slice/crio-a363234793984a65c5f50a9b15929d92329f548b3bf8e61ddb128d96e40b6b42 WatchSource:0}: Error finding container a363234793984a65c5f50a9b15929d92329f548b3bf8e61ddb128d96e40b6b42: Status 404 returned error can't find the container with id a363234793984a65c5f50a9b15929d92329f548b3bf8e61ddb128d96e40b6b42
Oct 30 03:11:35.905883 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:35.905844    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr" event={"ID":"46a4cb6f-f62d-4c7e-b4d4-3aaeb1e2c3de","Type":"ContainerStarted","Data":"a363234793984a65c5f50a9b15929d92329f548b3bf8e61ddb128d96e40b6b42"}
Oct 30 03:11:36.001519 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.001448    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h"]
Oct 30 03:11:36.005991 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.005970    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h"
Oct 30 03:11:36.026381 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.026357    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"perses-operator-service-cert\""
Oct 30 03:11:36.026491 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.026397    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"perses-operator-dockercfg-wrkv7\""
Oct 30 03:11:36.026581 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.026565    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:11:36.026694 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.026680    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"kube-root-ca.crt\""
Oct 30 03:11:36.041530 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.041511    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h"]
Oct 30 03:11:36.151907 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.151885    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zqlt\" (UniqueName: \"kubernetes.io/projected/2fe23053-6ef7-4485-aff2-b2d0a59473c9-kube-api-access-7zqlt\") pod \"perses-operator-7d5964d9b7-sz68h\" (UID: \"2fe23053-6ef7-4485-aff2-b2d0a59473c9\") " pod="openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h"
Oct 30 03:11:36.152029 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.151917    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe23053-6ef7-4485-aff2-b2d0a59473c9-openshift-service-ca\") pod \"perses-operator-7d5964d9b7-sz68h\" (UID: \"2fe23053-6ef7-4485-aff2-b2d0a59473c9\") " pod="openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h"
Oct 30 03:11:36.152029 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.151940    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fe23053-6ef7-4485-aff2-b2d0a59473c9-webhook-cert\") pod \"perses-operator-7d5964d9b7-sz68h\" (UID: \"2fe23053-6ef7-4485-aff2-b2d0a59473c9\") " pod="openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h"
Oct 30 03:11:36.152029 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.151964    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2fe23053-6ef7-4485-aff2-b2d0a59473c9-apiservice-cert\") pod \"perses-operator-7d5964d9b7-sz68h\" (UID: \"2fe23053-6ef7-4485-aff2-b2d0a59473c9\") " pod="openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h"
Oct 30 03:11:36.253399 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.253334    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zqlt\" (UniqueName: \"kubernetes.io/projected/2fe23053-6ef7-4485-aff2-b2d0a59473c9-kube-api-access-7zqlt\") pod \"perses-operator-7d5964d9b7-sz68h\" (UID: \"2fe23053-6ef7-4485-aff2-b2d0a59473c9\") " pod="openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h"
Oct 30 03:11:36.253399 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.253376    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe23053-6ef7-4485-aff2-b2d0a59473c9-openshift-service-ca\") pod \"perses-operator-7d5964d9b7-sz68h\" (UID: \"2fe23053-6ef7-4485-aff2-b2d0a59473c9\") " pod="openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h"
Oct 30 03:11:36.253570 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.253411    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fe23053-6ef7-4485-aff2-b2d0a59473c9-webhook-cert\") pod \"perses-operator-7d5964d9b7-sz68h\" (UID: \"2fe23053-6ef7-4485-aff2-b2d0a59473c9\") " pod="openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h"
Oct 30 03:11:36.253570 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.253450    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2fe23053-6ef7-4485-aff2-b2d0a59473c9-apiservice-cert\") pod \"perses-operator-7d5964d9b7-sz68h\" (UID: \"2fe23053-6ef7-4485-aff2-b2d0a59473c9\") " pod="openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h"
Oct 30 03:11:36.253992 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.253968    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe23053-6ef7-4485-aff2-b2d0a59473c9-openshift-service-ca\") pod \"perses-operator-7d5964d9b7-sz68h\" (UID: \"2fe23053-6ef7-4485-aff2-b2d0a59473c9\") " pod="openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h"
Oct 30 03:11:36.256006 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.255986    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2fe23053-6ef7-4485-aff2-b2d0a59473c9-apiservice-cert\") pod \"perses-operator-7d5964d9b7-sz68h\" (UID: \"2fe23053-6ef7-4485-aff2-b2d0a59473c9\") " pod="openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h"
Oct 30 03:11:36.256094 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.256029    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fe23053-6ef7-4485-aff2-b2d0a59473c9-webhook-cert\") pod \"perses-operator-7d5964d9b7-sz68h\" (UID: \"2fe23053-6ef7-4485-aff2-b2d0a59473c9\") " pod="openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h"
Oct 30 03:11:36.276750 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.276722    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zqlt\" (UniqueName: \"kubernetes.io/projected/2fe23053-6ef7-4485-aff2-b2d0a59473c9-kube-api-access-7zqlt\") pod \"perses-operator-7d5964d9b7-sz68h\" (UID: \"2fe23053-6ef7-4485-aff2-b2d0a59473c9\") " pod="openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h"
Oct 30 03:11:36.318625 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.318601    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h"
Oct 30 03:11:36.652911 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.652726    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h"]
Oct 30 03:11:36.656166 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:11:36.656128    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe23053_6ef7_4485_aff2_b2d0a59473c9.slice/crio-b378cf1f622714f68e50e3ed45bf505fdaa198e89e70f8b8103677e205bf342c WatchSource:0}: Error finding container b378cf1f622714f68e50e3ed45bf505fdaa198e89e70f8b8103677e205bf342c: Status 404 returned error can't find the container with id b378cf1f622714f68e50e3ed45bf505fdaa198e89e70f8b8103677e205bf342c
Oct 30 03:11:36.913042 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:36.912967    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h" event={"ID":"2fe23053-6ef7-4485-aff2-b2d0a59473c9","Type":"ContainerStarted","Data":"b378cf1f622714f68e50e3ed45bf505fdaa198e89e70f8b8103677e205bf342c"}
Oct 30 03:11:37.224147 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.224058    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:37.344911 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.344870    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5"]
Oct 30 03:11:37.372172 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.372141    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5"
Oct 30 03:11:37.378772 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.378742    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5"]
Oct 30 03:11:37.389821 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.389782    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-tempo-operator\"/\"webhook-server-cert\""
Oct 30 03:11:37.390782 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.390747    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-tempo-operator\"/\"tempo-operator-controller-service-cert\""
Oct 30 03:11:37.391022 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.391001    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-tempo-operator\"/\"kube-root-ca.crt\""
Oct 30 03:11:37.391100 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.391067    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-tempo-operator\"/\"tempo-operator-metrics\""
Oct 30 03:11:37.391237 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.391220    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-tempo-operator\"/\"tempo-operator-controller-manager-dockercfg-95mtc\""
Oct 30 03:11:37.396258 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.391798    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-tempo-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:11:37.450109 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.450074    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:37.463249 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.463213    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ljlj\" (UniqueName: \"kubernetes.io/projected/a6248f81-0e11-4a32-a5aa-983196043673-kube-api-access-4ljlj\") pod \"tempo-operator-controller-65b7645cf8-c5xp5\" (UID: \"a6248f81-0e11-4a32-a5aa-983196043673\") " pod="openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5"
Oct 30 03:11:37.463393 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.463268    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6248f81-0e11-4a32-a5aa-983196043673-tempo-operator-metrics-cert\") pod \"tempo-operator-controller-65b7645cf8-c5xp5\" (UID: \"a6248f81-0e11-4a32-a5aa-983196043673\") " pod="openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5"
Oct 30 03:11:37.463393 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.463299    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a6248f81-0e11-4a32-a5aa-983196043673-webhook-cert\") pod \"tempo-operator-controller-65b7645cf8-c5xp5\" (UID: \"a6248f81-0e11-4a32-a5aa-983196043673\") " pod="openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5"
Oct 30 03:11:37.463393 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.463342    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a6248f81-0e11-4a32-a5aa-983196043673-apiservice-cert\") pod \"tempo-operator-controller-65b7645cf8-c5xp5\" (UID: \"a6248f81-0e11-4a32-a5aa-983196043673\") " pod="openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5"
Oct 30 03:11:37.564722 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.564651    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ljlj\" (UniqueName: \"kubernetes.io/projected/a6248f81-0e11-4a32-a5aa-983196043673-kube-api-access-4ljlj\") pod \"tempo-operator-controller-65b7645cf8-c5xp5\" (UID: \"a6248f81-0e11-4a32-a5aa-983196043673\") " pod="openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5"
Oct 30 03:11:37.564722 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.564704    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6248f81-0e11-4a32-a5aa-983196043673-tempo-operator-metrics-cert\") pod \"tempo-operator-controller-65b7645cf8-c5xp5\" (UID: \"a6248f81-0e11-4a32-a5aa-983196043673\") " pod="openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5"
Oct 30 03:11:37.564937 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.564743    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a6248f81-0e11-4a32-a5aa-983196043673-webhook-cert\") pod \"tempo-operator-controller-65b7645cf8-c5xp5\" (UID: \"a6248f81-0e11-4a32-a5aa-983196043673\") " pod="openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5"
Oct 30 03:11:37.564937 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.564823    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a6248f81-0e11-4a32-a5aa-983196043673-apiservice-cert\") pod \"tempo-operator-controller-65b7645cf8-c5xp5\" (UID: \"a6248f81-0e11-4a32-a5aa-983196043673\") " pod="openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5"
Oct 30 03:11:37.568267 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.568241    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a6248f81-0e11-4a32-a5aa-983196043673-apiservice-cert\") pod \"tempo-operator-controller-65b7645cf8-c5xp5\" (UID: \"a6248f81-0e11-4a32-a5aa-983196043673\") " pod="openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5"
Oct 30 03:11:37.568889 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.568868    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a6248f81-0e11-4a32-a5aa-983196043673-webhook-cert\") pod \"tempo-operator-controller-65b7645cf8-c5xp5\" (UID: \"a6248f81-0e11-4a32-a5aa-983196043673\") " pod="openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5"
Oct 30 03:11:37.580570 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.580541    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6248f81-0e11-4a32-a5aa-983196043673-tempo-operator-metrics-cert\") pod \"tempo-operator-controller-65b7645cf8-c5xp5\" (UID: \"a6248f81-0e11-4a32-a5aa-983196043673\") " pod="openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5"
Oct 30 03:11:37.611308 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.611251    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ljlj\" (UniqueName: \"kubernetes.io/projected/a6248f81-0e11-4a32-a5aa-983196043673-kube-api-access-4ljlj\") pod \"tempo-operator-controller-65b7645cf8-c5xp5\" (UID: \"a6248f81-0e11-4a32-a5aa-983196043673\") " pod="openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5"
Oct 30 03:11:37.648488 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.648419    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:37.690560 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.690384    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5"
Oct 30 03:11:37.860418 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:37.860385    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:38.047536 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:38.047502    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5"]
Oct 30 03:11:38.050433 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:11:38.050399    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6248f81_0e11_4a32_a5aa_983196043673.slice/crio-daabd0ff6191372fa51cc3abf090acbc14534061889ac05a891157caf2b2e041 WatchSource:0}: Error finding container daabd0ff6191372fa51cc3abf090acbc14534061889ac05a891157caf2b2e041: Status 404 returned error can't find the container with id daabd0ff6191372fa51cc3abf090acbc14534061889ac05a891157caf2b2e041
Oct 30 03:11:38.924680 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:38.924645    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5" event={"ID":"a6248f81-0e11-4a32-a5aa-983196043673","Type":"ContainerStarted","Data":"daabd0ff6191372fa51cc3abf090acbc14534061889ac05a891157caf2b2e041"}
Oct 30 03:11:38.926591 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:38.926556    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr" event={"ID":"46a4cb6f-f62d-4c7e-b4d4-3aaeb1e2c3de","Type":"ContainerStarted","Data":"3ff60ae6118cf168d11c6a8e6351b65f5f94fb6a08f526fa8500b7d0bd715885"}
Oct 30 03:11:38.928721 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:38.928364    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h" event={"ID":"2fe23053-6ef7-4485-aff2-b2d0a59473c9","Type":"ContainerStarted","Data":"400c478ad6ded0b8275ccb3f94e0099aa0b918d3ce45a619ec6e375fca499ed4"}
Oct 30 03:11:38.928721 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:38.928519    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h"
Oct 30 03:11:39.042995 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:39.042951    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-6sgcr" podStartSLOduration=1.5285791720000002 podStartE2EDuration="4.042936337s" totalImagesPullingTime="2.514357165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:39.029128299 +0000 UTC m=+1779.678176721" watchObservedRunningTime="2026-10-30 03:11:39.042936337 +0000 UTC m=+1779.691984759"
Oct 30 03:11:39.134163 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:39.134112    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h" podStartSLOduration=2.407741541 podStartE2EDuration="4.134071473s" totalImagesPullingTime="1.726329932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:39.131870346 +0000 UTC m=+1779.780918780" watchObservedRunningTime="2026-10-30 03:11:39.134071473 +0000 UTC m=+1779.783119886"
Oct 30 03:11:39.933610 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:39.933579    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5" event={"ID":"a6248f81-0e11-4a32-a5aa-983196043673","Type":"ContainerStarted","Data":"fff8dae7a8571de9264df20902aad24e0509a35548043f6b5b3ec5529bd5a693"}
Oct 30 03:11:39.933922 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:39.933902    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5"
Oct 30 03:11:40.021932 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:40.021890    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5" podStartSLOduration=1.23728206 podStartE2EDuration="3.0218793s" totalImagesPullingTime="1.78459724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:40.021165786 +0000 UTC m=+1780.670214207" watchObservedRunningTime="2026-10-30 03:11:40.0218793 +0000 UTC m=+1780.670927720"
Oct 30 03:11:42.242278 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:42.242246    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:42.461037 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:42.461004    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:42.737902 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:42.737876    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:42.983823 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:42.983797    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:43.135933 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:43.135897    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:49.936290 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:49.936256    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-cluster-observability-operator/perses-operator-7d5964d9b7-sz68h"
Oct 30 03:11:50.941701 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:50.941666    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-tempo-operator/tempo-operator-controller-65b7645cf8-c5xp5"
Oct 30 03:11:52.911263 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:52.911170    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:53.953313 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:11:53.953282    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:11:58.472371 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:58.472333    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:58.716435 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:58.716403    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:58.837705 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:58.837683    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:59.529299 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:59.529267    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:59.612317 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:59.612291    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp"]
Oct 30 03:11:59.617722 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:59.617706    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp"
Oct 30 03:11:59.641461 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:59.641440    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:11:59.644791 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:59.644758    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-metrics\""
Oct 30 03:11:59.646632 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:59.646606    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:11:59.655325 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:59.655307    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:11:59.660277 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:59.660252    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp"]
Oct 30 03:11:59.660400 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:59.660328    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:11:59.721021 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:59.720999    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:59.727116 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:59.727097    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c7e7c42-cf7d-48b0-91e1-b7e2b1286785-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-7c764765c6-9q6dp\" (UID: \"7c7e7c42-cf7d-48b0-91e1-b7e2b1286785\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp"
Oct 30 03:11:59.727200 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:59.727154    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f77tz\" (UniqueName: \"kubernetes.io/projected/7c7e7c42-cf7d-48b0-91e1-b7e2b1286785-kube-api-access-f77tz\") pod \"opentelemetry-operator-controller-manager-7c764765c6-9q6dp\" (UID: \"7c7e7c42-cf7d-48b0-91e1-b7e2b1286785\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp"
Oct 30 03:11:59.827752 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:59.827727    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f77tz\" (UniqueName: \"kubernetes.io/projected/7c7e7c42-cf7d-48b0-91e1-b7e2b1286785-kube-api-access-f77tz\") pod \"opentelemetry-operator-controller-manager-7c764765c6-9q6dp\" (UID: \"7c7e7c42-cf7d-48b0-91e1-b7e2b1286785\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp"
Oct 30 03:11:59.827841 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:59.827765    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c7e7c42-cf7d-48b0-91e1-b7e2b1286785-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-7c764765c6-9q6dp\" (UID: \"7c7e7c42-cf7d-48b0-91e1-b7e2b1286785\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp"
Oct 30 03:11:59.830238 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:59.830218    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c7e7c42-cf7d-48b0-91e1-b7e2b1286785-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-7c764765c6-9q6dp\" (UID: \"7c7e7c42-cf7d-48b0-91e1-b7e2b1286785\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp"
Oct 30 03:11:59.849889 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:59.849860    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f77tz\" (UniqueName: \"kubernetes.io/projected/7c7e7c42-cf7d-48b0-91e1-b7e2b1286785-kube-api-access-f77tz\") pod \"opentelemetry-operator-controller-manager-7c764765c6-9q6dp\" (UID: \"7c7e7c42-cf7d-48b0-91e1-b7e2b1286785\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp"
Oct 30 03:11:59.929727 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:11:59.929704    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp"
Oct 30 03:12:00.142065 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:00.141976    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:00.270946 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:00.270924    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp"]
Oct 30 03:12:00.271568 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:12:00.271545    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7e7c42_cf7d_48b0_91e1_b7e2b1286785.slice/crio-0d92605a0a2197570d841185cab6171f5d6d305e272238bd16d0b0cc454692e9 WatchSource:0}: Error finding container 0d92605a0a2197570d841185cab6171f5d6d305e272238bd16d0b0cc454692e9: Status 404 returned error can't find the container with id 0d92605a0a2197570d841185cab6171f5d6d305e272238bd16d0b0cc454692e9
Oct 30 03:12:00.330145 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:00.330124    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:01.022622 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:01.022580    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp" event={"ID":"7c7e7c42-cf7d-48b0-91e1-b7e2b1286785","Type":"ContainerStarted","Data":"1313a3189f66675a834fb5168ec726b1b87144a6ffe7681d1a0179a90a964685"}
Oct 30 03:12:01.023085 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:01.022646    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp" event={"ID":"7c7e7c42-cf7d-48b0-91e1-b7e2b1286785","Type":"ContainerStarted","Data":"0d92605a0a2197570d841185cab6171f5d6d305e272238bd16d0b0cc454692e9"}
Oct 30 03:12:01.023085 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:01.022677    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp"
Oct 30 03:12:01.129274 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:01.129227    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp" podStartSLOduration=2.1292136 podStartE2EDuration="2.1292136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:01.115618773 +0000 UTC m=+1801.764667194" watchObservedRunningTime="2026-10-30 03:12:01.1292136 +0000 UTC m=+1801.778262021"
Oct 30 03:12:02.469595 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:02.469557    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp"]
Oct 30 03:12:03.031822 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:03.031774    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp" podUID="7c7e7c42-cf7d-48b0-91e1-b7e2b1286785" containerName="manager" containerID="cri-o://1313a3189f66675a834fb5168ec726b1b87144a6ffe7681d1a0179a90a964685" gracePeriod=10
Oct 30 03:12:03.276825 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:03.276803    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp"
Oct 30 03:12:03.357873 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:03.357850    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c7e7c42-cf7d-48b0-91e1-b7e2b1286785-opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c7e7c42-cf7d-48b0-91e1-b7e2b1286785-opentelemetry-operator-metrics-cert\") pod \"7c7e7c42-cf7d-48b0-91e1-b7e2b1286785\" (UID: \"7c7e7c42-cf7d-48b0-91e1-b7e2b1286785\") "
Oct 30 03:12:03.358004 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:03.357888    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7e7c42-cf7d-48b0-91e1-b7e2b1286785-kube-api-access-f77tz\" (UniqueName: \"kubernetes.io/projected/7c7e7c42-cf7d-48b0-91e1-b7e2b1286785-kube-api-access-f77tz\") pod \"7c7e7c42-cf7d-48b0-91e1-b7e2b1286785\" (UID: \"7c7e7c42-cf7d-48b0-91e1-b7e2b1286785\") "
Oct 30 03:12:03.360144 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:03.360113    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7e7c42-cf7d-48b0-91e1-b7e2b1286785-kube-api-access-f77tz" pod "7c7e7c42-cf7d-48b0-91e1-b7e2b1286785" (UID: "7c7e7c42-cf7d-48b0-91e1-b7e2b1286785"). InnerVolumeSpecName "kube-api-access-f77tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:03.360268 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:03.360160    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c7e7c42-cf7d-48b0-91e1-b7e2b1286785-opentelemetry-operator-metrics-cert" pod "7c7e7c42-cf7d-48b0-91e1-b7e2b1286785" (UID: "7c7e7c42-cf7d-48b0-91e1-b7e2b1286785"). InnerVolumeSpecName "opentelemetry-operator-metrics-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:12:03.458607 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:03.458577    4960 reconciler_common.go:299] "Volume detached for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c7e7c42-cf7d-48b0-91e1-b7e2b1286785-opentelemetry-operator-metrics-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:12:03.458709 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:03.458606    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f77tz\" (UniqueName: \"kubernetes.io/projected/7c7e7c42-cf7d-48b0-91e1-b7e2b1286785-kube-api-access-f77tz\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:12:04.038654 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:04.038620    4960 generic.go:356] "Generic (PLEG): container finished" podID="7c7e7c42-cf7d-48b0-91e1-b7e2b1286785" containerID="1313a3189f66675a834fb5168ec726b1b87144a6ffe7681d1a0179a90a964685" exitCode=0
Oct 30 03:12:04.039018 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:04.038686    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp"
Oct 30 03:12:04.039018 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:04.038722    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp" event={"ID":"7c7e7c42-cf7d-48b0-91e1-b7e2b1286785","Type":"ContainerDied","Data":"1313a3189f66675a834fb5168ec726b1b87144a6ffe7681d1a0179a90a964685"}
Oct 30 03:12:04.039018 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:04.038788    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp" event={"ID":"7c7e7c42-cf7d-48b0-91e1-b7e2b1286785","Type":"ContainerDied","Data":"0d92605a0a2197570d841185cab6171f5d6d305e272238bd16d0b0cc454692e9"}
Oct 30 03:12:04.039018 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:04.038812    4960 scope.go:122] "RemoveContainer" containerID="1313a3189f66675a834fb5168ec726b1b87144a6ffe7681d1a0179a90a964685"
Oct 30 03:12:04.048427 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:04.048408    4960 scope.go:122] "RemoveContainer" containerID="1313a3189f66675a834fb5168ec726b1b87144a6ffe7681d1a0179a90a964685"
Oct 30 03:12:04.048754 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:12:04.048729    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1313a3189f66675a834fb5168ec726b1b87144a6ffe7681d1a0179a90a964685\": container with ID starting with 1313a3189f66675a834fb5168ec726b1b87144a6ffe7681d1a0179a90a964685 not found: ID does not exist" containerID="1313a3189f66675a834fb5168ec726b1b87144a6ffe7681d1a0179a90a964685"
Oct 30 03:12:04.048800 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:04.048766    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1313a3189f66675a834fb5168ec726b1b87144a6ffe7681d1a0179a90a964685"} err="failed to get container status \"1313a3189f66675a834fb5168ec726b1b87144a6ffe7681d1a0179a90a964685\": rpc error: code = NotFound desc = could not find container \"1313a3189f66675a834fb5168ec726b1b87144a6ffe7681d1a0179a90a964685\": container with ID starting with 1313a3189f66675a834fb5168ec726b1b87144a6ffe7681d1a0179a90a964685 not found: ID does not exist"
Oct 30 03:12:04.119104 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:04.119078    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp"]
Oct 30 03:12:04.132412 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:04.132388    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7c764765c6-9q6dp"]
Oct 30 03:12:05.503276 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:05.503234    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7e7c42-cf7d-48b0-91e1-b7e2b1286785" path="/var/lib/kubelet/pods/7c7e7c42-cf7d-48b0-91e1-b7e2b1286785/volumes"
Oct 30 03:12:11.038141 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:11.038108    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:11.237633 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:11.237599    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:11.568904 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:11.568862    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:11.683840 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:11.683807    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:12.247918 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:12.247884    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:13.639604 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:13.639571    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:13.798513 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:13.798484    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:15.316741 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:15.316709    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:15.464958 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:15.464923    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:15.974794 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:15.974759    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:16.131577 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:16.131543    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:16.414066 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:16.414037    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:16.574076 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:16.574048    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:18.207656 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:18.207612    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:18.379619 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:18.379585    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:20.428380 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:20.428342    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:20.589916 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:20.589876    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:22.626010 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:22.625974    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:22.757852 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:22.757817    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:23.957234 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:12:23.957169    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:12:33.315220 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:33.315174    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:33.468948 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:33.468917    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:38.371646 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.371607    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:38.403147 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.403113    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"]
Oct 30 03:12:38.403465 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.403452    4960 cpu_manager.go:414] "containerMap: removing container" podUID="7c7e7c42-cf7d-48b0-91e1-b7e2b1286785" containerName="manager"
Oct 30 03:12:38.403508 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.403467    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7e7c42-cf7d-48b0-91e1-b7e2b1286785" containerName="manager"
Oct 30 03:12:38.403541 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.403529    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7e7c42-cf7d-48b0-91e1-b7e2b1286785" containerName="manager"
Oct 30 03:12:38.405500 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.405484    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"
Oct 30 03:12:38.436709 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.436683    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-wfj9m\""
Oct 30 03:12:38.436968 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.436939    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\""
Oct 30 03:12:38.437770 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.437751    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"]
Oct 30 03:12:38.439454 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.439432    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\""
Oct 30 03:12:38.439539 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.439529    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:12:38.439602 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.439591    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:12:38.492267 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.492232    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:38.516695 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.516671    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a33dd179-4d86-4c85-81ff-faf0e3d6272f-config\") pod \"korrel8r-6644769df5-lwdxq\" (UID: \"a33dd179-4d86-4c85-81ff-faf0e3d6272f\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"
Oct 30 03:12:38.516792 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.516718    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a33dd179-4d86-4c85-81ff-faf0e3d6272f-tls\") pod \"korrel8r-6644769df5-lwdxq\" (UID: \"a33dd179-4d86-4c85-81ff-faf0e3d6272f\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"
Oct 30 03:12:38.516847 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.516829    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzxck\" (UniqueName: \"kubernetes.io/projected/a33dd179-4d86-4c85-81ff-faf0e3d6272f-kube-api-access-tzxck\") pod \"korrel8r-6644769df5-lwdxq\" (UID: \"a33dd179-4d86-4c85-81ff-faf0e3d6272f\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"
Oct 30 03:12:38.607114 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.607087    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:38.617135 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.617112    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzxck\" (UniqueName: \"kubernetes.io/projected/a33dd179-4d86-4c85-81ff-faf0e3d6272f-kube-api-access-tzxck\") pod \"korrel8r-6644769df5-lwdxq\" (UID: \"a33dd179-4d86-4c85-81ff-faf0e3d6272f\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"
Oct 30 03:12:38.617267 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.617166    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a33dd179-4d86-4c85-81ff-faf0e3d6272f-config\") pod \"korrel8r-6644769df5-lwdxq\" (UID: \"a33dd179-4d86-4c85-81ff-faf0e3d6272f\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"
Oct 30 03:12:38.617267 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.617219    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a33dd179-4d86-4c85-81ff-faf0e3d6272f-tls\") pod \"korrel8r-6644769df5-lwdxq\" (UID: \"a33dd179-4d86-4c85-81ff-faf0e3d6272f\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"
Oct 30 03:12:38.618003 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.617983    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a33dd179-4d86-4c85-81ff-faf0e3d6272f-config\") pod \"korrel8r-6644769df5-lwdxq\" (UID: \"a33dd179-4d86-4c85-81ff-faf0e3d6272f\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"
Oct 30 03:12:38.619920 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.619896    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a33dd179-4d86-4c85-81ff-faf0e3d6272f-tls\") pod \"korrel8r-6644769df5-lwdxq\" (UID: \"a33dd179-4d86-4c85-81ff-faf0e3d6272f\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"
Oct 30 03:12:38.638226 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.638159    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzxck\" (UniqueName: \"kubernetes.io/projected/a33dd179-4d86-4c85-81ff-faf0e3d6272f-kube-api-access-tzxck\") pod \"korrel8r-6644769df5-lwdxq\" (UID: \"a33dd179-4d86-4c85-81ff-faf0e3d6272f\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"
Oct 30 03:12:38.717904 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.717878    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"
Oct 30 03:12:38.734360 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:38.734337    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:39.019436 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.019413    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"]
Oct 30 03:12:39.021549 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:12:39.021524    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33dd179_4d86_4c85_81ff_faf0e3d6272f.slice/crio-78ca983058b553ebfca837e275ed5e3734f74676f49bc41eebcf7dc443d0510c WatchSource:0}: Error finding container 78ca983058b553ebfca837e275ed5e3734f74676f49bc41eebcf7dc443d0510c: Status 404 returned error can't find the container with id 78ca983058b553ebfca837e275ed5e3734f74676f49bc41eebcf7dc443d0510c
Oct 30 03:12:39.200054 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.200016    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq" event={"ID":"a33dd179-4d86-4c85-81ff-faf0e3d6272f","Type":"ContainerStarted","Data":"78ca983058b553ebfca837e275ed5e3734f74676f49bc41eebcf7dc443d0510c"}
Oct 30 03:12:39.289739 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.289661    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:39.334979 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.334948    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:12:39.339808 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.339783    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:12:39.361978 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.361948    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:12:39.421582 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.421556    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:39.424688 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.424664    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/aad2a338-0e3c-4aba-ab6f-1a1202075383-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"aad2a338-0e3c-4aba-ab6f-1a1202075383\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:12:39.424788 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.424708    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/aad2a338-0e3c-4aba-ab6f-1a1202075383-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"aad2a338-0e3c-4aba-ab6f-1a1202075383\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:12:39.424788 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.424779    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/aad2a338-0e3c-4aba-ab6f-1a1202075383-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"aad2a338-0e3c-4aba-ab6f-1a1202075383\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:12:39.424862 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.424843    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsq9w\" (UniqueName: \"kubernetes.io/projected/aad2a338-0e3c-4aba-ab6f-1a1202075383-kube-api-access-fsq9w\") pod \"data-science-collector-collector-1\" (UID: \"aad2a338-0e3c-4aba-ab6f-1a1202075383\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:12:39.426024 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.426009    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:12:39.426114 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.426022    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:12:39.426114 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.426030    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-lqnhh\""
Oct 30 03:12:39.426605 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.426587    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\""
Oct 30 03:12:39.525907 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.525878    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsq9w\" (UniqueName: \"kubernetes.io/projected/aad2a338-0e3c-4aba-ab6f-1a1202075383-kube-api-access-fsq9w\") pod \"data-science-collector-collector-1\" (UID: \"aad2a338-0e3c-4aba-ab6f-1a1202075383\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:12:39.526030 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.525947    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/aad2a338-0e3c-4aba-ab6f-1a1202075383-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"aad2a338-0e3c-4aba-ab6f-1a1202075383\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:12:39.526030 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.525977    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/aad2a338-0e3c-4aba-ab6f-1a1202075383-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"aad2a338-0e3c-4aba-ab6f-1a1202075383\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:12:39.526030 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.526003    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/aad2a338-0e3c-4aba-ab6f-1a1202075383-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"aad2a338-0e3c-4aba-ab6f-1a1202075383\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:12:39.526759 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.526723    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/aad2a338-0e3c-4aba-ab6f-1a1202075383-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"aad2a338-0e3c-4aba-ab6f-1a1202075383\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:12:39.528918 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.528895    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/aad2a338-0e3c-4aba-ab6f-1a1202075383-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"aad2a338-0e3c-4aba-ab6f-1a1202075383\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:12:39.529611 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.529594    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/aad2a338-0e3c-4aba-ab6f-1a1202075383-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"aad2a338-0e3c-4aba-ab6f-1a1202075383\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:12:39.547034 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.546987    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsq9w\" (UniqueName: \"kubernetes.io/projected/aad2a338-0e3c-4aba-ab6f-1a1202075383-kube-api-access-fsq9w\") pod \"data-science-collector-collector-1\" (UID: \"aad2a338-0e3c-4aba-ab6f-1a1202075383\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:12:39.584173 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.584145    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:39.655352 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.655330    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:12:39.980028 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:39.980002    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:12:39.981704 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:12:39.981678    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad2a338_0e3c_4aba_ab6f_1a1202075383.slice/crio-46c1f195d2f0eb3874728a445ab35b00125a65d078632aafe37c9fef7841894a WatchSource:0}: Error finding container 46c1f195d2f0eb3874728a445ab35b00125a65d078632aafe37c9fef7841894a: Status 404 returned error can't find the container with id 46c1f195d2f0eb3874728a445ab35b00125a65d078632aafe37c9fef7841894a
Oct 30 03:12:40.206947 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:40.206907    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"aad2a338-0e3c-4aba-ab6f-1a1202075383","Type":"ContainerStarted","Data":"46c1f195d2f0eb3874728a445ab35b00125a65d078632aafe37c9fef7841894a"}
Oct 30 03:12:40.947659 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:40.947558    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:41.062538 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:41.062501    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:41.173927 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:41.173889    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:41.905057 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:41.905023    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:42.021577 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:42.021538    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:42.220915 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:42.220832    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq" event={"ID":"a33dd179-4d86-4c85-81ff-faf0e3d6272f","Type":"ContainerStarted","Data":"11b0e3ef7c809f1bf2ca41bc50b2b5a254a3b6775badfb6402eb95fa59452d90"}
Oct 30 03:12:42.221079 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:42.220980    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"
Oct 30 03:12:42.276950 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:42.276895    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq" podStartSLOduration=2.031687428 podStartE2EDuration="4.276878869s" totalImagesPullingTime="2.245191441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:42.26196698 +0000 UTC m=+1842.911015403" watchObservedRunningTime="2026-10-30 03:12:42.276878869 +0000 UTC m=+1842.925927293"
Oct 30 03:12:42.508020 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:42.507914    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:43.265146 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:43.265093    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:43.384600 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:43.384507    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:43.807120 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:43.807089    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:12:43.807238 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:43.807155    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:12:44.195840 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:44.195803    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:44.232535 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:44.232509    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"aad2a338-0e3c-4aba-ab6f-1a1202075383","Type":"ContainerStarted","Data":"d9dfe18ca399aa14eae2e243a14d3f40c675d04ee0a6ca0bc8cf6103e922fac0"}
Oct 30 03:12:44.275243 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:44.275165    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.453164342 podStartE2EDuration="5.275141978s" totalImagesPullingTime="3.821977636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:44.274525048 +0000 UTC m=+1844.923573468" watchObservedRunningTime="2026-10-30 03:12:44.275141978 +0000 UTC m=+1844.924190399"
Oct 30 03:12:52.967615 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:52.967578    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:53.229652 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:53.229578    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"
Oct 30 03:12:53.953595 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:12:53.953562    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:12:57.428680 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:57.428643    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"]
Oct 30 03:12:57.429072 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:57.428820    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq" podUID="a33dd179-4d86-4c85-81ff-faf0e3d6272f" containerName="korrel8r" containerID="cri-o://11b0e3ef7c809f1bf2ca41bc50b2b5a254a3b6775badfb6402eb95fa59452d90" gracePeriod=30
Oct 30 03:12:57.672357 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:57.672331    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"
Oct 30 03:12:57.763365 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:57.763285    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a33dd179-4d86-4c85-81ff-faf0e3d6272f-tls\" (UniqueName: \"kubernetes.io/secret/a33dd179-4d86-4c85-81ff-faf0e3d6272f-tls\") pod \"a33dd179-4d86-4c85-81ff-faf0e3d6272f\" (UID: \"a33dd179-4d86-4c85-81ff-faf0e3d6272f\") "
Oct 30 03:12:57.763524 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:57.763363    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33dd179-4d86-4c85-81ff-faf0e3d6272f-config\" (UniqueName: \"kubernetes.io/configmap/a33dd179-4d86-4c85-81ff-faf0e3d6272f-config\") pod \"a33dd179-4d86-4c85-81ff-faf0e3d6272f\" (UID: \"a33dd179-4d86-4c85-81ff-faf0e3d6272f\") "
Oct 30 03:12:57.763740 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:57.763711    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33dd179-4d86-4c85-81ff-faf0e3d6272f-config" pod "a33dd179-4d86-4c85-81ff-faf0e3d6272f" (UID: "a33dd179-4d86-4c85-81ff-faf0e3d6272f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:57.765546 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:57.765527    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a33dd179-4d86-4c85-81ff-faf0e3d6272f-tls" pod "a33dd179-4d86-4c85-81ff-faf0e3d6272f" (UID: "a33dd179-4d86-4c85-81ff-faf0e3d6272f"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:12:57.851393 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:57.851362    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:12:57.851568 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:57.851543    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="aad2a338-0e3c-4aba-ab6f-1a1202075383" containerName="otc-container" containerID="cri-o://d9dfe18ca399aa14eae2e243a14d3f40c675d04ee0a6ca0bc8cf6103e922fac0" gracePeriod=30
Oct 30 03:12:57.863860 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:57.863828    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33dd179-4d86-4c85-81ff-faf0e3d6272f-kube-api-access-tzxck\" (UniqueName: \"kubernetes.io/projected/a33dd179-4d86-4c85-81ff-faf0e3d6272f-kube-api-access-tzxck\") pod \"a33dd179-4d86-4c85-81ff-faf0e3d6272f\" (UID: \"a33dd179-4d86-4c85-81ff-faf0e3d6272f\") "
Oct 30 03:12:57.865050 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:57.865030    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a33dd179-4d86-4c85-81ff-faf0e3d6272f-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:12:57.865216 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:57.865174    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a33dd179-4d86-4c85-81ff-faf0e3d6272f-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:12:57.869358 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:57.869330    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33dd179-4d86-4c85-81ff-faf0e3d6272f-kube-api-access-tzxck" pod "a33dd179-4d86-4c85-81ff-faf0e3d6272f" (UID: "a33dd179-4d86-4c85-81ff-faf0e3d6272f"). InnerVolumeSpecName "kube-api-access-tzxck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:57.920911 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:57.920883    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:57.966693 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:57.966670    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzxck\" (UniqueName: \"kubernetes.io/projected/a33dd179-4d86-4c85-81ff-faf0e3d6272f-kube-api-access-tzxck\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:12:58.029529 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.029499    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:58.083082 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.083061    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:12:58.168084 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.168054    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad2a338-0e3c-4aba-ab6f-1a1202075383-kube-api-access-fsq9w\" (UniqueName: \"kubernetes.io/projected/aad2a338-0e3c-4aba-ab6f-1a1202075383-kube-api-access-fsq9w\") pod \"aad2a338-0e3c-4aba-ab6f-1a1202075383\" (UID: \"aad2a338-0e3c-4aba-ab6f-1a1202075383\") "
Oct 30 03:12:58.168241 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.168108    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aad2a338-0e3c-4aba-ab6f-1a1202075383-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/aad2a338-0e3c-4aba-ab6f-1a1202075383-monitoring-tls-certs\") pod \"aad2a338-0e3c-4aba-ab6f-1a1202075383\" (UID: \"aad2a338-0e3c-4aba-ab6f-1a1202075383\") "
Oct 30 03:12:58.168241 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.168135    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aad2a338-0e3c-4aba-ab6f-1a1202075383-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/aad2a338-0e3c-4aba-ab6f-1a1202075383-otlp-tls-certs\") pod \"aad2a338-0e3c-4aba-ab6f-1a1202075383\" (UID: \"aad2a338-0e3c-4aba-ab6f-1a1202075383\") "
Oct 30 03:12:58.168354 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.168237    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad2a338-0e3c-4aba-ab6f-1a1202075383-otc-internal\" (UniqueName: \"kubernetes.io/configmap/aad2a338-0e3c-4aba-ab6f-1a1202075383-otc-internal\") pod \"aad2a338-0e3c-4aba-ab6f-1a1202075383\" (UID: \"aad2a338-0e3c-4aba-ab6f-1a1202075383\") "
Oct 30 03:12:58.168675 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.168649    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad2a338-0e3c-4aba-ab6f-1a1202075383-otc-internal" pod "aad2a338-0e3c-4aba-ab6f-1a1202075383" (UID: "aad2a338-0e3c-4aba-ab6f-1a1202075383"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:58.169481 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.169445    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:58.170523 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.170496    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aad2a338-0e3c-4aba-ab6f-1a1202075383-otlp-tls-certs" pod "aad2a338-0e3c-4aba-ab6f-1a1202075383" (UID: "aad2a338-0e3c-4aba-ab6f-1a1202075383"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:12:58.170614 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.170549    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad2a338-0e3c-4aba-ab6f-1a1202075383-kube-api-access-fsq9w" pod "aad2a338-0e3c-4aba-ab6f-1a1202075383" (UID: "aad2a338-0e3c-4aba-ab6f-1a1202075383"). InnerVolumeSpecName "kube-api-access-fsq9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:58.170730 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.170702    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aad2a338-0e3c-4aba-ab6f-1a1202075383-monitoring-tls-certs" pod "aad2a338-0e3c-4aba-ab6f-1a1202075383" (UID: "aad2a338-0e3c-4aba-ab6f-1a1202075383"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:12:58.269119 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.269095    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsq9w\" (UniqueName: \"kubernetes.io/projected/aad2a338-0e3c-4aba-ab6f-1a1202075383-kube-api-access-fsq9w\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:12:58.269119 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.269119    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/aad2a338-0e3c-4aba-ab6f-1a1202075383-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:12:58.269288 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.269128    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/aad2a338-0e3c-4aba-ab6f-1a1202075383-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:12:58.269288 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.269137    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/aad2a338-0e3c-4aba-ab6f-1a1202075383-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:12:58.294128 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.294059    4960 generic.go:356] "Generic (PLEG): container finished" podID="a33dd179-4d86-4c85-81ff-faf0e3d6272f" containerID="11b0e3ef7c809f1bf2ca41bc50b2b5a254a3b6775badfb6402eb95fa59452d90" exitCode=0
Oct 30 03:12:58.294128 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.294105    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"
Oct 30 03:12:58.294319 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.294145    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq" event={"ID":"a33dd179-4d86-4c85-81ff-faf0e3d6272f","Type":"ContainerDied","Data":"11b0e3ef7c809f1bf2ca41bc50b2b5a254a3b6775badfb6402eb95fa59452d90"}
Oct 30 03:12:58.294319 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.294223    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-lwdxq" event={"ID":"a33dd179-4d86-4c85-81ff-faf0e3d6272f","Type":"ContainerDied","Data":"78ca983058b553ebfca837e275ed5e3734f74676f49bc41eebcf7dc443d0510c"}
Oct 30 03:12:58.294319 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.294248    4960 scope.go:122] "RemoveContainer" containerID="11b0e3ef7c809f1bf2ca41bc50b2b5a254a3b6775badfb6402eb95fa59452d90"
Oct 30 03:12:58.295541 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.295516    4960 generic.go:356] "Generic (PLEG): container finished" podID="aad2a338-0e3c-4aba-ab6f-1a1202075383" containerID="d9dfe18ca399aa14eae2e243a14d3f40c675d04ee0a6ca0bc8cf6103e922fac0" exitCode=0
Oct 30 03:12:58.295634 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.295542    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"aad2a338-0e3c-4aba-ab6f-1a1202075383","Type":"ContainerDied","Data":"d9dfe18ca399aa14eae2e243a14d3f40c675d04ee0a6ca0bc8cf6103e922fac0"}
Oct 30 03:12:58.295634 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.295565    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:12:58.295634 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.295592    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"aad2a338-0e3c-4aba-ab6f-1a1202075383","Type":"ContainerDied","Data":"46c1f195d2f0eb3874728a445ab35b00125a65d078632aafe37c9fef7841894a"}
Oct 30 03:12:58.309575 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.309238    4960 scope.go:122] "RemoveContainer" containerID="11b0e3ef7c809f1bf2ca41bc50b2b5a254a3b6775badfb6402eb95fa59452d90"
Oct 30 03:12:58.309725 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:12:58.309596    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b0e3ef7c809f1bf2ca41bc50b2b5a254a3b6775badfb6402eb95fa59452d90\": container with ID starting with 11b0e3ef7c809f1bf2ca41bc50b2b5a254a3b6775badfb6402eb95fa59452d90 not found: ID does not exist" containerID="11b0e3ef7c809f1bf2ca41bc50b2b5a254a3b6775badfb6402eb95fa59452d90"
Oct 30 03:12:58.309725 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.309633    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b0e3ef7c809f1bf2ca41bc50b2b5a254a3b6775badfb6402eb95fa59452d90"} err="failed to get container status \"11b0e3ef7c809f1bf2ca41bc50b2b5a254a3b6775badfb6402eb95fa59452d90\": rpc error: code = NotFound desc = could not find container \"11b0e3ef7c809f1bf2ca41bc50b2b5a254a3b6775badfb6402eb95fa59452d90\": container with ID starting with 11b0e3ef7c809f1bf2ca41bc50b2b5a254a3b6775badfb6402eb95fa59452d90 not found: ID does not exist"
Oct 30 03:12:58.309725 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.309657    4960 scope.go:122] "RemoveContainer" containerID="d9dfe18ca399aa14eae2e243a14d3f40c675d04ee0a6ca0bc8cf6103e922fac0"
Oct 30 03:12:58.318801 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.318779    4960 scope.go:122] "RemoveContainer" containerID="d9dfe18ca399aa14eae2e243a14d3f40c675d04ee0a6ca0bc8cf6103e922fac0"
Oct 30 03:12:58.319069 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:12:58.319049    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9dfe18ca399aa14eae2e243a14d3f40c675d04ee0a6ca0bc8cf6103e922fac0\": container with ID starting with d9dfe18ca399aa14eae2e243a14d3f40c675d04ee0a6ca0bc8cf6103e922fac0 not found: ID does not exist" containerID="d9dfe18ca399aa14eae2e243a14d3f40c675d04ee0a6ca0bc8cf6103e922fac0"
Oct 30 03:12:58.319161 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.319075    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9dfe18ca399aa14eae2e243a14d3f40c675d04ee0a6ca0bc8cf6103e922fac0"} err="failed to get container status \"d9dfe18ca399aa14eae2e243a14d3f40c675d04ee0a6ca0bc8cf6103e922fac0\": rpc error: code = NotFound desc = could not find container \"d9dfe18ca399aa14eae2e243a14d3f40c675d04ee0a6ca0bc8cf6103e922fac0\": container with ID starting with d9dfe18ca399aa14eae2e243a14d3f40c675d04ee0a6ca0bc8cf6103e922fac0 not found: ID does not exist"
Oct 30 03:12:58.353206 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.353150    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:12:58.357565 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.357541    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:12:58.385627 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.385602    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:58.393860 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.393836    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"]
Oct 30 03:12:58.399452 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.399429    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-lwdxq"]
Oct 30 03:12:58.493006 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:58.492975    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:59.502826 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:59.502796    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a33dd179-4d86-4c85-81ff-faf0e3d6272f" path="/var/lib/kubelet/pods/a33dd179-4d86-4c85-81ff-faf0e3d6272f/volumes"
Oct 30 03:12:59.503218 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:12:59.503166    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad2a338-0e3c-4aba-ab6f-1a1202075383" path="/var/lib/kubelet/pods/aad2a338-0e3c-4aba-ab6f-1a1202075383/volumes"
Oct 30 03:13:02.751775 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:02.751734    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:02.882536 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:02.882494    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:03.009670 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.009594    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:03.041223 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.041193    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"]
Oct 30 03:13:03.041518 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.041506    4960 cpu_manager.go:414] "containerMap: removing container" podUID="a33dd179-4d86-4c85-81ff-faf0e3d6272f" containerName="korrel8r"
Oct 30 03:13:03.041560 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.041520    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33dd179-4d86-4c85-81ff-faf0e3d6272f" containerName="korrel8r"
Oct 30 03:13:03.041560 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.041539    4960 cpu_manager.go:414] "containerMap: removing container" podUID="aad2a338-0e3c-4aba-ab6f-1a1202075383" containerName="otc-container"
Oct 30 03:13:03.041560 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.041544    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad2a338-0e3c-4aba-ab6f-1a1202075383" containerName="otc-container"
Oct 30 03:13:03.041659 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.041597    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad2a338-0e3c-4aba-ab6f-1a1202075383" containerName="otc-container"
Oct 30 03:13:03.041659 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.041608    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33dd179-4d86-4c85-81ff-faf0e3d6272f" containerName="korrel8r"
Oct 30 03:13:03.044516 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.044500    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:03.065478 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.065451    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:13:03.066830 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.066805    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"]
Oct 30 03:13:03.079028 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.078984    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:13:03.079154 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.079025    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 03:13:03.079154 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.079057    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\""
Oct 30 03:13:03.079154 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.079065    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 03:13:03.079154 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.079065    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\""
Oct 30 03:13:03.079154 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.079101    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-47skz\""
Oct 30 03:13:03.103712 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.103683    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-kfvxs\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:03.103801 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.103737    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-kfvxs\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:03.103885 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.103867    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-kfvxs\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:03.103946 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.103909    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bljmz\" (UniqueName: \"kubernetes.io/projected/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-kube-api-access-bljmz\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-kfvxs\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:03.104008 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.103943    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-kfvxs\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:03.177722 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.177699    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:03.189306 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.189283    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"]
Oct 30 03:13:03.191644 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.191624    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"
Oct 30 03:13:03.204995 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.204975    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-kfvxs\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:03.205081 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.205008    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bljmz\" (UniqueName: \"kubernetes.io/projected/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-kube-api-access-bljmz\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-kfvxs\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:03.205081 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.205028    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-kfvxs\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:03.205081 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.205058    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-kfvxs\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:03.205278 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.205105    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-kfvxs\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:03.205278 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:13:03.205171    4960 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found
Oct 30 03:13:03.205278 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:13:03.205249    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-client-cert podName:d2ecd98d-b049-4ce5-84f6-0546aed2d0f0 nodeName:}" failed. No retries permitted until 2026-10-30 03:13:03.705228947 +0000 UTC m=+1864.354277348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-kfvxs" (UID: "d2ecd98d-b049-4ce5-84f6-0546aed2d0f0") : secret "prometheus-operated-tls" not found
Oct 30 03:13:03.205772 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.205751    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-kfvxs\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:03.207699 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.207682    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-kfvxs\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:03.207773 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.207747    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-kfvxs\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:03.212598 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.212580    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"]
Oct 30 03:13:03.226816 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.226793    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bljmz\" (UniqueName: \"kubernetes.io/projected/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-kube-api-access-bljmz\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-kfvxs\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:03.266089 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.266032    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\""
Oct 30 03:13:03.266089 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.266062    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-g98fm\""
Oct 30 03:13:03.266277 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.266105    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\""
Oct 30 03:13:03.303068 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.303036    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:03.306221 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.306204    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/04fff530-1d0e-4e48-8d16-57bdba3a0ede-tls\") pod \"korrel8r-5cbcc89b8f-wb2rw\" (UID: \"04fff530-1d0e-4e48-8d16-57bdba3a0ede\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"
Oct 30 03:13:03.306298 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.306238    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92vfb\" (UniqueName: \"kubernetes.io/projected/04fff530-1d0e-4e48-8d16-57bdba3a0ede-kube-api-access-92vfb\") pod \"korrel8r-5cbcc89b8f-wb2rw\" (UID: \"04fff530-1d0e-4e48-8d16-57bdba3a0ede\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"
Oct 30 03:13:03.306298 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.306270    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04fff530-1d0e-4e48-8d16-57bdba3a0ede-config\") pod \"korrel8r-5cbcc89b8f-wb2rw\" (UID: \"04fff530-1d0e-4e48-8d16-57bdba3a0ede\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"
Oct 30 03:13:03.406902 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.406878    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92vfb\" (UniqueName: \"kubernetes.io/projected/04fff530-1d0e-4e48-8d16-57bdba3a0ede-kube-api-access-92vfb\") pod \"korrel8r-5cbcc89b8f-wb2rw\" (UID: \"04fff530-1d0e-4e48-8d16-57bdba3a0ede\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"
Oct 30 03:13:03.407005 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.406929    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04fff530-1d0e-4e48-8d16-57bdba3a0ede-config\") pod \"korrel8r-5cbcc89b8f-wb2rw\" (UID: \"04fff530-1d0e-4e48-8d16-57bdba3a0ede\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"
Oct 30 03:13:03.407070 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.407014    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/04fff530-1d0e-4e48-8d16-57bdba3a0ede-tls\") pod \"korrel8r-5cbcc89b8f-wb2rw\" (UID: \"04fff530-1d0e-4e48-8d16-57bdba3a0ede\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"
Oct 30 03:13:03.407680 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.407664    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04fff530-1d0e-4e48-8d16-57bdba3a0ede-config\") pod \"korrel8r-5cbcc89b8f-wb2rw\" (UID: \"04fff530-1d0e-4e48-8d16-57bdba3a0ede\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"
Oct 30 03:13:03.409336 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.409314    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/04fff530-1d0e-4e48-8d16-57bdba3a0ede-tls\") pod \"korrel8r-5cbcc89b8f-wb2rw\" (UID: \"04fff530-1d0e-4e48-8d16-57bdba3a0ede\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"
Oct 30 03:13:03.429418 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.429398    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92vfb\" (UniqueName: \"kubernetes.io/projected/04fff530-1d0e-4e48-8d16-57bdba3a0ede-kube-api-access-92vfb\") pod \"korrel8r-5cbcc89b8f-wb2rw\" (UID: \"04fff530-1d0e-4e48-8d16-57bdba3a0ede\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"
Oct 30 03:13:03.481012 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.480989    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:03.506246 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.506228    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"
Oct 30 03:13:03.695848 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.695819    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:03.708856 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.708831    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-kfvxs\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:03.709005 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:13:03.708987    4960 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found
Oct 30 03:13:03.709079 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:13:03.709064    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-client-cert podName:d2ecd98d-b049-4ce5-84f6-0546aed2d0f0 nodeName:}" failed. No retries permitted until 2026-10-30 03:13:04.709044167 +0000 UTC m=+1865.358092568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-kfvxs" (UID: "d2ecd98d-b049-4ce5-84f6-0546aed2d0f0") : secret "prometheus-operated-tls" not found
Oct 30 03:13:03.798882 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.798862    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"]
Oct 30 03:13:03.801211 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:13:03.801161    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04fff530_1d0e_4e48_8d16_57bdba3a0ede.slice/crio-8736460cb10bbe65a7f9f2054ade62b6d7bac3234cb845ecf4f5a57836f62a61 WatchSource:0}: Error finding container 8736460cb10bbe65a7f9f2054ade62b6d7bac3234cb845ecf4f5a57836f62a61: Status 404 returned error can't find the container with id 8736460cb10bbe65a7f9f2054ade62b6d7bac3234cb845ecf4f5a57836f62a61
Oct 30 03:13:03.821929 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:03.821904    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:04.084394 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.084365    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:04.324668 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.324633    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw" event={"ID":"04fff530-1d0e-4e48-8d16-57bdba3a0ede","Type":"ContainerStarted","Data":"978437718cea2ff216f3e59353d1b882c5fc64804d017e7e360cbfd294286f8b"}
Oct 30 03:13:04.324801 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.324692    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw" event={"ID":"04fff530-1d0e-4e48-8d16-57bdba3a0ede","Type":"ContainerStarted","Data":"8736460cb10bbe65a7f9f2054ade62b6d7bac3234cb845ecf4f5a57836f62a61"}
Oct 30 03:13:04.324801 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.324731    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"
Oct 30 03:13:04.366255 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.366130    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw" podStartSLOduration=1.366117658 podStartE2EDuration="1.366117658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:04.366061677 +0000 UTC m=+1865.015110109" watchObservedRunningTime="2026-10-30 03:13:04.366117658 +0000 UTC m=+1865.015166073"
Oct 30 03:13:04.648989 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.648911    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:13:04.659699 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.659673    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:04.676690 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.676665    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:13:04.685298 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.685273    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:13:04.688392 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.688369    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:13:04.688548 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.688411    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:13:04.688898 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.688881    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:13:04.691227 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.691154    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:13:04.716895 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.716865    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:04.717076 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.716907    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:04.717076 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.716933    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:04.717076 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.717034    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:04.717076 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.717067    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c785c29d-afbb-4d76-bbe5-e8791e58c432-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:04.717315 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.717109    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-kfvxs\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:04.717315 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.717213    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqnrf\" (UniqueName: \"kubernetes.io/projected/c785c29d-afbb-4d76-bbe5-e8791e58c432-kube-api-access-lqnrf\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:04.719915 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.719891    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-kfvxs\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:04.723616 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.723584    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:04.819668 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.819629    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:04.820131 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.819682    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:04.820131 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.819731    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:04.820131 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.819772    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c785c29d-afbb-4d76-bbe5-e8791e58c432-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:04.820131 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.819824    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqnrf\" (UniqueName: \"kubernetes.io/projected/c785c29d-afbb-4d76-bbe5-e8791e58c432-kube-api-access-lqnrf\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:04.820131 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.819886    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:04.820131 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:13:04.819914    4960 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found
Oct 30 03:13:04.820131 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:13:04.820043    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-data-science-collector-ta-client-cert podName:c785c29d-afbb-4d76-bbe5-e8791e58c432 nodeName:}" failed. No retries permitted until 2026-10-30 03:13:05.32001755 +0000 UTC m=+1865.969065964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-data-science-collector-ta-client-cert") pod "data-science-collector-collector-0" (UID: "c785c29d-afbb-4d76-bbe5-e8791e58c432") : secret "data-science-collector-ta-client-cert" not found
Oct 30 03:13:04.820979 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.820951    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c785c29d-afbb-4d76-bbe5-e8791e58c432-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:04.822865 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.822542    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:04.822865 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.822607    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:04.822865 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.822824    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:04.846822 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.846801    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqnrf\" (UniqueName: \"kubernetes.io/projected/c785c29d-afbb-4d76-bbe5-e8791e58c432-kube-api-access-lqnrf\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:04.850647 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.850621    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:04.855613 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.855594    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:04.988368 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:04.988340    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:05.164336 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:05.164313    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"]
Oct 30 03:13:05.166231 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:13:05.166200    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ecd98d_b049_4ce5_84f6_0546aed2d0f0.slice/crio-9e125f95766066f484482743c0a9731f8693f165ff985639997df02ca6803993 WatchSource:0}: Error finding container 9e125f95766066f484482743c0a9731f8693f165ff985639997df02ca6803993: Status 404 returned error can't find the container with id 9e125f95766066f484482743c0a9731f8693f165ff985639997df02ca6803993
Oct 30 03:13:05.324013 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:05.323985    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:05.324146 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:13:05.324130    4960 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found
Oct 30 03:13:05.324227 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:13:05.324216    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-data-science-collector-ta-client-cert podName:c785c29d-afbb-4d76-bbe5-e8791e58c432 nodeName:}" failed. No retries permitted until 2026-10-30 03:13:06.324200042 +0000 UTC m=+1866.973248454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-data-science-collector-ta-client-cert") pod "data-science-collector-collector-0" (UID: "c785c29d-afbb-4d76-bbe5-e8791e58c432") : secret "data-science-collector-ta-client-cert" not found
Oct 30 03:13:05.329623 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:05.329597    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs" event={"ID":"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0","Type":"ContainerStarted","Data":"9e125f95766066f484482743c0a9731f8693f165ff985639997df02ca6803993"}
Oct 30 03:13:06.332860 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:06.332821    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:06.335785 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:06.335760    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:06.476920 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:06.476889    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:06.833591 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:06.833553    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:13:06.838231 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:13:06.838153    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc785c29d_afbb_4d76_bbe5_e8791e58c432.slice/crio-d3f815254378718f18838732f387aee2931f279094d24010d8c973b4422c2c20 WatchSource:0}: Error finding container d3f815254378718f18838732f387aee2931f279094d24010d8c973b4422c2c20: Status 404 returned error can't find the container with id d3f815254378718f18838732f387aee2931f279094d24010d8c973b4422c2c20
Oct 30 03:13:06.841121 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:06.841087    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:13:06.841255 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:06.841159    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:13:07.340988 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:07.340963    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c785c29d-afbb-4d76-bbe5-e8791e58c432","Type":"ContainerStarted","Data":"e5aacbf7dfebbcb154203823fe761bab066890bac98375baa9aa70806efb3409"}
Oct 30 03:13:07.341328 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:07.341018    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c785c29d-afbb-4d76-bbe5-e8791e58c432","Type":"ContainerStarted","Data":"d3f815254378718f18838732f387aee2931f279094d24010d8c973b4422c2c20"}
Oct 30 03:13:07.392066 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:07.392023    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=3.392009629 podStartE2EDuration="3.392009629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:07.390909799 +0000 UTC m=+1868.039958275" watchObservedRunningTime="2026-10-30 03:13:07.392009629 +0000 UTC m=+1868.041058083"
Oct 30 03:13:07.548083 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:07.548049    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:07.790958 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:07.790920    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:07.914007 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:07.913977    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:08.196680 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.196652    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 03:13:08.199466 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.199450    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.224941 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.223378    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\""
Oct 30 03:13:08.224941 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.223396    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\""
Oct 30 03:13:08.224941 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.223549    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-zzqx2\""
Oct 30 03:13:08.238479 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.238455    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 03:13:08.248978 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.248951    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:08.251982 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.251956    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1b42492-cd77-4565-913b-94670d1be346-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.252105 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.251997    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/d1b42492-cd77-4565-913b-94670d1be346-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.252259 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.252207    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.252358 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.252341    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.253775 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.253747    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1b42492-cd77-4565-913b-94670d1be346-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.253924 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.253807    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.253924 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.253838    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzx5h\" (UniqueName: \"kubernetes.io/projected/d1b42492-cd77-4565-913b-94670d1be346-kube-api-access-lzx5h\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.326560 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.326537    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\""
Oct 30 03:13:08.326724 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.326699    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\""
Oct 30 03:13:08.347687 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.347657    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs" event={"ID":"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0","Type":"ContainerStarted","Data":"5f056fd53f0269d043d0801306e31a3ccc895b49981193b2f5278553bb5f5683"}
Oct 30 03:13:08.348032 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.347872    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:08.355394 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.355373    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1b42492-cd77-4565-913b-94670d1be346-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.355456 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.355401    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/d1b42492-cd77-4565-913b-94670d1be346-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.355456 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.355446    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.355551 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.355484    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.355551 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.355514    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1b42492-cd77-4565-913b-94670d1be346-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.355551 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.355542    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.355703 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.355569    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzx5h\" (UniqueName: \"kubernetes.io/projected/d1b42492-cd77-4565-913b-94670d1be346-kube-api-access-lzx5h\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.355906 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.355859    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/d1b42492-cd77-4565-913b-94670d1be346-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.358127 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.358100    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1b42492-cd77-4565-913b-94670d1be346-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.358252 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.358129    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.358337 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.358252    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.358488 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.358472    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1b42492-cd77-4565-913b-94670d1be346-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.358610 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.358593    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.367794 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.367768    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:08.375812 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.375789    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzx5h\" (UniqueName: \"kubernetes.io/projected/d1b42492-cd77-4565-913b-94670d1be346-kube-api-access-lzx5h\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.397780 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.397735    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs" podStartSLOduration=3.242870718 podStartE2EDuration="5.397720187s" totalImagesPullingTime="2.154849469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:08.397292032 +0000 UTC m=+1869.046340453" watchObservedRunningTime="2026-10-30 03:13:08.397720187 +0000 UTC m=+1869.046768607"
Oct 30 03:13:08.494932 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.494875    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:08.513957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.513936    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:08.829784 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:08.829762    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 03:13:08.831827 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:13:08.831804    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b42492_cd77_4565_913b_94670d1be346.slice/crio-a2483d70b840c0f9470fb3908bca2aa9d2b62ef8c4b2b1f99ed737fede9d4744 WatchSource:0}: Error finding container a2483d70b840c0f9470fb3908bca2aa9d2b62ef8c4b2b1f99ed737fede9d4744: Status 404 returned error can't find the container with id a2483d70b840c0f9470fb3908bca2aa9d2b62ef8c4b2b1f99ed737fede9d4744
Oct 30 03:13:09.353870 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:09.353830    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"d1b42492-cd77-4565-913b-94670d1be346","Type":"ContainerStarted","Data":"a2483d70b840c0f9470fb3908bca2aa9d2b62ef8c4b2b1f99ed737fede9d4744"}
Oct 30 03:13:10.588416 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:10.588382    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:10.704815 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:10.704741    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:11.296799 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.296769    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:13:11.299533 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.299516    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.317419 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.317391    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 03:13:11.330488 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.330462    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 03:13:11.330488 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.330484    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 03:13:11.330488 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.330461    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 03:13:11.330675 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.330495    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 03:13:11.330675 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.330516    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 03:13:11.330675 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.330533    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-jz5xx\""
Oct 30 03:13:11.330675 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.330534    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 03:13:11.330675 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.330580    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 03:13:11.331924 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.331906    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:13:11.364438 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.364408    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"d1b42492-cd77-4565-913b-94670d1be346","Type":"ContainerStarted","Data":"bc627a37430812e902275cf518392edaadf57747b423929d22628464856db1dd"}
Oct 30 03:13:11.389540 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.389513    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.389655 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.389553    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.389655 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.389574    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.389655 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.389646    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a07bc058-5869-4242-8c84-a9858cd7eae3-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.389798 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.389674    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs4h7\" (UniqueName: \"kubernetes.io/projected/a07bc058-5869-4242-8c84-a9858cd7eae3-kube-api-access-xs4h7\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.389798 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.389699    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.389872 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.389810    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.389872 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.389838    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a07bc058-5869-4242-8c84-a9858cd7eae3-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.389955 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.389873    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.389955 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.389930    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.390031 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.389974    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.390081 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.390061    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.459564 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.459538    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:11.490823 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.490789    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a07bc058-5869-4242-8c84-a9858cd7eae3-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.490958 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.490837    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.490958 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.490914    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.491079 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.490957    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.491079 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.490993    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.491206 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.491110    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.491721 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.491622    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.491721 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.491487    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.491721 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.491755    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.492145 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.492114    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a07bc058-5869-4242-8c84-a9858cd7eae3-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.492815 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.492784    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs4h7\" (UniqueName: \"kubernetes.io/projected/a07bc058-5869-4242-8c84-a9858cd7eae3-kube-api-access-xs4h7\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.493162 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.493130    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.494645 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.492924    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.494645 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.493834    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.494645 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.494296    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a07bc058-5869-4242-8c84-a9858cd7eae3-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.494645 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.494279    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.494645 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.492530    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.494645 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.494485    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.495502 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.495477    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a07bc058-5869-4242-8c84-a9858cd7eae3-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.495596 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.495486    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.495951 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.495928    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.496760 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.496718    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.497548 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.497522    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.519551 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.519525    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs4h7\" (UniqueName: \"kubernetes.io/projected/a07bc058-5869-4242-8c84-a9858cd7eae3-kube-api-access-xs4h7\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:11.704746 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:11.704718    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:14.356390 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:14.356365    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:13:14.379333 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:14.379298    4960 generic.go:356] "Generic (PLEG): container finished" podID="d1b42492-cd77-4565-913b-94670d1be346" containerID="bc627a37430812e902275cf518392edaadf57747b423929d22628464856db1dd" exitCode=0
Oct 30 03:13:14.379492 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:14.379364    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"d1b42492-cd77-4565-913b-94670d1be346","Type":"ContainerDied","Data":"bc627a37430812e902275cf518392edaadf57747b423929d22628464856db1dd"}
Oct 30 03:13:15.332948 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:15.332915    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"
Oct 30 03:13:16.393867 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:16.393835    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"d1b42492-cd77-4565-913b-94670d1be346","Type":"ContainerStarted","Data":"fc49999a8678c4fbf80581cdbcadf453acc789bdeec2eb1f59e1c3967b067546"}
Oct 30 03:13:16.740157 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:16.740074    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:16.771321 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:16.771288    4960 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:16.840752 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:16.840724    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:16.840875 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:16.840850    4960 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:16.856718 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:16.856693    4960 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") DevicePath \"csi-6f8c8e4c1326657251c30594036033694cadf25e97408e4f9d9da436f78f3fa4\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:17.043389 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:17.043363    4960 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/8c516923ea0339579ebf808cef81b9e0aaf55e5b619e2c6187025fa3b2b51f7f/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:17.070676 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:17.070653    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:17.314797 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:17.314736    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:17.401259 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:17.401213    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"d1b42492-cd77-4565-913b-94670d1be346","Type":"ContainerStarted","Data":"6101bf1446e1b0bdf794364a18d68215f9f33693ecdb00f774acd5dcd02c4826"}
Oct 30 03:13:17.401636 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:17.401371    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:17.448720 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:17.448681    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.173507743 podStartE2EDuration="9.448669038s" totalImagesPullingTime="2.687736885s" totalInitContainerRuntime="3.58742441s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:17.448166749 +0000 UTC m=+1878.097215167" watchObservedRunningTime="2026-10-30 03:13:17.448669038 +0000 UTC m=+1878.097717460"
Oct 30 03:13:17.623812 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:17.623778    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:13:17.624844 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:13:17.624820    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07bc058_5869_4242_8c84_a9858cd7eae3.slice/crio-2d45a61e8ded1e7162027bbc9e3c60107f20dcffc18ad57d1eb5ebbef87cc42e WatchSource:0}: Error finding container 2d45a61e8ded1e7162027bbc9e3c60107f20dcffc18ad57d1eb5ebbef87cc42e: Status 404 returned error can't find the container with id 2d45a61e8ded1e7162027bbc9e3c60107f20dcffc18ad57d1eb5ebbef87cc42e
Oct 30 03:13:18.408550 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:18.408512    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a07bc058-5869-4242-8c84-a9858cd7eae3","Type":"ContainerStarted","Data":"0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba"}
Oct 30 03:13:18.408973 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:18.408585    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a07bc058-5869-4242-8c84-a9858cd7eae3","Type":"ContainerStarted","Data":"2d45a61e8ded1e7162027bbc9e3c60107f20dcffc18ad57d1eb5ebbef87cc42e"}
Oct 30 03:13:18.410811 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:18.410786    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:13:21.424154 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:21.424120    4960 generic.go:356] "Generic (PLEG): container finished" podID="a07bc058-5869-4242-8c84-a9858cd7eae3" containerID="0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba" exitCode=0
Oct 30 03:13:21.424527 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:21.424205    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a07bc058-5869-4242-8c84-a9858cd7eae3","Type":"ContainerDied","Data":"0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba"}
Oct 30 03:13:23.955286 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:13:23.955250    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:13:25.446857 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:25.446820    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a07bc058-5869-4242-8c84-a9858cd7eae3","Type":"ContainerStarted","Data":"a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8"}
Oct 30 03:13:25.447259 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:25.446926    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a07bc058-5869-4242-8c84-a9858cd7eae3","Type":"ContainerStarted","Data":"6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b"}
Oct 30 03:13:27.458138 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:27.458104    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a07bc058-5869-4242-8c84-a9858cd7eae3","Type":"ContainerStarted","Data":"50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50"}
Oct 30 03:13:27.550642 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:27.550599    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=11.387876721 podStartE2EDuration="19.550584901s" totalImagesPullingTime="4.975978406s" totalInitContainerRuntime="3.186729774s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:13:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:27.536005001 +0000 UTC m=+1888.185053422" watchObservedRunningTime="2026-10-30 03:13:27.550584901 +0000 UTC m=+1888.199633321"
Oct 30 03:13:32.209645 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:32.209612    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:32.314932 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:32.314901    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:32.315063 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:32.314985    4960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:32.320282 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:32.320263    4960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:32.332369 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:32.332347    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:32.485387 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:32.485322    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:13:42.799021 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:42.798987    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:42.907726 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:42.907696    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:43.104433 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:43.104401    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:43.223495 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:43.223467    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:44.721311 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:44.721274    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:44.860948 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:44.860915    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:47.857488 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:47.857447    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:13:47.858056 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:47.857682    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="c785c29d-afbb-4d76-bbe5-e8791e58c432" containerName="otc-container" containerID="cri-o://e5aacbf7dfebbcb154203823fe761bab066890bac98375baa9aa70806efb3409" gracePeriod=30
Oct 30 03:13:49.109203 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.109165    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.204062 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.203990    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c785c29d-afbb-4d76-bbe5-e8791e58c432-otc-internal\" (UniqueName: \"kubernetes.io/configmap/c785c29d-afbb-4d76-bbe5-e8791e58c432-otc-internal\") pod \"c785c29d-afbb-4d76-bbe5-e8791e58c432\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") "
Oct 30 03:13:49.204062 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.204027    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-tls-certs\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-tls-certs\") pod \"c785c29d-afbb-4d76-bbe5-e8791e58c432\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") "
Oct 30 03:13:49.204303 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.204070    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c785c29d-afbb-4d76-bbe5-e8791e58c432-kube-api-access-lqnrf\" (UniqueName: \"kubernetes.io/projected/c785c29d-afbb-4d76-bbe5-e8791e58c432-kube-api-access-lqnrf\") pod \"c785c29d-afbb-4d76-bbe5-e8791e58c432\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") "
Oct 30 03:13:49.204303 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.204104    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-monitoring-tls-certs\") pod \"c785c29d-afbb-4d76-bbe5-e8791e58c432\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") "
Oct 30 03:13:49.204303 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.204137    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-otlp-tls-certs\") pod \"c785c29d-afbb-4d76-bbe5-e8791e58c432\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") "
Oct 30 03:13:49.204303 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.204194    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-data-science-collector-ta-client-cert\") pod \"c785c29d-afbb-4d76-bbe5-e8791e58c432\" (UID: \"c785c29d-afbb-4d76-bbe5-e8791e58c432\") "
Oct 30 03:13:49.204544 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.204515    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c785c29d-afbb-4d76-bbe5-e8791e58c432-otc-internal" pod "c785c29d-afbb-4d76-bbe5-e8791e58c432" (UID: "c785c29d-afbb-4d76-bbe5-e8791e58c432"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:49.206569 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.206537    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-monitoring-tls-certs" pod "c785c29d-afbb-4d76-bbe5-e8791e58c432" (UID: "c785c29d-afbb-4d76-bbe5-e8791e58c432"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:49.206755 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.206730    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-data-science-collector-ta-client-cert" pod "c785c29d-afbb-4d76-bbe5-e8791e58c432" (UID: "c785c29d-afbb-4d76-bbe5-e8791e58c432"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:49.206834 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.206760    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-tls-certs" pod "c785c29d-afbb-4d76-bbe5-e8791e58c432" (UID: "c785c29d-afbb-4d76-bbe5-e8791e58c432"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:49.207079 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.207053    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c785c29d-afbb-4d76-bbe5-e8791e58c432-kube-api-access-lqnrf" pod "c785c29d-afbb-4d76-bbe5-e8791e58c432" (UID: "c785c29d-afbb-4d76-bbe5-e8791e58c432"). InnerVolumeSpecName "kube-api-access-lqnrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:49.207197 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.207080    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-otlp-tls-certs" pod "c785c29d-afbb-4d76-bbe5-e8791e58c432" (UID: "c785c29d-afbb-4d76-bbe5-e8791e58c432"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:49.310088 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.305727    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqnrf\" (UniqueName: \"kubernetes.io/projected/c785c29d-afbb-4d76-bbe5-e8791e58c432-kube-api-access-lqnrf\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:13:49.310088 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.305757    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:13:49.310088 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.305772    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:13:49.310088 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.305795    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:13:49.310088 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.305818    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c785c29d-afbb-4d76-bbe5-e8791e58c432-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:13:49.310088 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.305833    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c785c29d-afbb-4d76-bbe5-e8791e58c432-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:13:49.557278 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.557248    4960 generic.go:356] "Generic (PLEG): container finished" podID="c785c29d-afbb-4d76-bbe5-e8791e58c432" containerID="e5aacbf7dfebbcb154203823fe761bab066890bac98375baa9aa70806efb3409" exitCode=0
Oct 30 03:13:49.557382 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.557313    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.557382 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.557332    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c785c29d-afbb-4d76-bbe5-e8791e58c432","Type":"ContainerDied","Data":"e5aacbf7dfebbcb154203823fe761bab066890bac98375baa9aa70806efb3409"}
Oct 30 03:13:49.557460 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.557386    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c785c29d-afbb-4d76-bbe5-e8791e58c432","Type":"ContainerDied","Data":"d3f815254378718f18838732f387aee2931f279094d24010d8c973b4422c2c20"}
Oct 30 03:13:49.557460 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.557418    4960 scope.go:122] "RemoveContainer" containerID="e5aacbf7dfebbcb154203823fe761bab066890bac98375baa9aa70806efb3409"
Oct 30 03:13:49.566017 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.565994    4960 scope.go:122] "RemoveContainer" containerID="e5aacbf7dfebbcb154203823fe761bab066890bac98375baa9aa70806efb3409"
Oct 30 03:13:49.566345 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:13:49.566322    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5aacbf7dfebbcb154203823fe761bab066890bac98375baa9aa70806efb3409\": container with ID starting with e5aacbf7dfebbcb154203823fe761bab066890bac98375baa9aa70806efb3409 not found: ID does not exist" containerID="e5aacbf7dfebbcb154203823fe761bab066890bac98375baa9aa70806efb3409"
Oct 30 03:13:49.566432 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.566351    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5aacbf7dfebbcb154203823fe761bab066890bac98375baa9aa70806efb3409"} err="failed to get container status \"e5aacbf7dfebbcb154203823fe761bab066890bac98375baa9aa70806efb3409\": rpc error: code = NotFound desc = could not find container \"e5aacbf7dfebbcb154203823fe761bab066890bac98375baa9aa70806efb3409\": container with ID starting with e5aacbf7dfebbcb154203823fe761bab066890bac98375baa9aa70806efb3409 not found: ID does not exist"
Oct 30 03:13:49.612282 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.612263    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:13:49.625951 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.625934    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:13:49.653084 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.653063    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:13:49.653417 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.653405    4960 cpu_manager.go:414] "containerMap: removing container" podUID="c785c29d-afbb-4d76-bbe5-e8791e58c432" containerName="otc-container"
Oct 30 03:13:49.653466 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.653419    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c785c29d-afbb-4d76-bbe5-e8791e58c432" containerName="otc-container"
Oct 30 03:13:49.653500 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.653496    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="c785c29d-afbb-4d76-bbe5-e8791e58c432" containerName="otc-container"
Oct 30 03:13:49.658397 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.658381    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.696952 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.696292    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:13:49.697146 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.697118    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:13:49.697338 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.697252    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:13:49.697422 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.697365    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:13:49.697543 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.697524    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:13:49.698957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.698934    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:13:49.708410 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.708391    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/198cf335-0852-48ff-870a-d7ba6113edc9-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.708518 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.708420    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.708518 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.708457    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7ts\" (UniqueName: \"kubernetes.io/projected/198cf335-0852-48ff-870a-d7ba6113edc9-kube-api-access-qn7ts\") pod \"data-science-collector-collector-0\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.708641 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.708558    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.708641 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.708595    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.708641 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.708632    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.709300 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.709276    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:13:49.809782 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.809633    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7ts\" (UniqueName: \"kubernetes.io/projected/198cf335-0852-48ff-870a-d7ba6113edc9-kube-api-access-qn7ts\") pod \"data-science-collector-collector-0\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.809782 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.809679    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.809782 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.809701    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.809782 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.809739    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.811341 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.810268    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/198cf335-0852-48ff-870a-d7ba6113edc9-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.811341 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.810313    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.811531 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.811378    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/198cf335-0852-48ff-870a-d7ba6113edc9-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.812979 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.812955    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.813091 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.812954    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.813091 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.813075    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.813091 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.813080    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.829792 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.829770    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7ts\" (UniqueName: \"kubernetes.io/projected/198cf335-0852-48ff-870a-d7ba6113edc9-kube-api-access-qn7ts\") pod \"data-science-collector-collector-0\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:49.971247 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:49.971223    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:13:50.270271 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:50.270248    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:13:50.272230 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:13:50.272205    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198cf335_0852_48ff_870a_d7ba6113edc9.slice/crio-74ec6f9994f86fbbd804bc6d7f2eaf156bd0303d75defaaa52194b5246074d81 WatchSource:0}: Error finding container 74ec6f9994f86fbbd804bc6d7f2eaf156bd0303d75defaaa52194b5246074d81: Status 404 returned error can't find the container with id 74ec6f9994f86fbbd804bc6d7f2eaf156bd0303d75defaaa52194b5246074d81
Oct 30 03:13:50.274567 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:50.274536    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:13:50.274671 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:50.274608    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:13:50.565022 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:50.564949    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"198cf335-0852-48ff-870a-d7ba6113edc9","Type":"ContainerStarted","Data":"db4d5072ad24a96f17637b4a82c37e6964d94750d95b32965ce517ec453aa996"}
Oct 30 03:13:50.565022 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:50.564992    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"198cf335-0852-48ff-870a-d7ba6113edc9","Type":"ContainerStarted","Data":"74ec6f9994f86fbbd804bc6d7f2eaf156bd0303d75defaaa52194b5246074d81"}
Oct 30 03:13:50.609579 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:50.609537    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.609524664 podStartE2EDuration="1.609524664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:50.608933369 +0000 UTC m=+1911.257981791" watchObservedRunningTime="2026-10-30 03:13:50.609524664 +0000 UTC m=+1911.258573122"
Oct 30 03:13:51.502803 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:51.502767    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c785c29d-afbb-4d76-bbe5-e8791e58c432" path="/var/lib/kubelet/pods/c785c29d-afbb-4d76-bbe5-e8791e58c432/volumes"
Oct 30 03:13:53.071318 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:53.071282    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:53.953863 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:13:53.953829    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:13:56.847463 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:56.847432    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:56.962312 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:13:56.962282    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:12.496258 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:14:12.496168    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:12.602448 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:14:12.602419    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:23.953516 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:14:23.953481    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:14:33.195257 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:14:33.195226    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:53.200324 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:14:53.200292    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:53.953677 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:14:53.953645    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:15:08.896042 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:08.896004    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:09.005362 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.005331    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:09.197300 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.197219    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:09.224204 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.224163    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"]
Oct 30 03:15:09.227608 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.227588    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"
Oct 30 03:15:09.248251 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.248221    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"]
Oct 30 03:15:09.323330 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.323301    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtfrm\" (UniqueName: \"kubernetes.io/projected/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-kube-api-access-wtfrm\") pod \"korrel8r-66d6f7f696-khq4q\" (UID: \"261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"
Oct 30 03:15:09.323520 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.323348    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-config\") pod \"korrel8r-66d6f7f696-khq4q\" (UID: \"261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"
Oct 30 03:15:09.323520 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.323371    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-tls\") pod \"korrel8r-66d6f7f696-khq4q\" (UID: \"261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"
Oct 30 03:15:09.424832 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.424804    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtfrm\" (UniqueName: \"kubernetes.io/projected/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-kube-api-access-wtfrm\") pod \"korrel8r-66d6f7f696-khq4q\" (UID: \"261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"
Oct 30 03:15:09.425007 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.424862    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-config\") pod \"korrel8r-66d6f7f696-khq4q\" (UID: \"261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"
Oct 30 03:15:09.425086 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.425065    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-tls\") pod \"korrel8r-66d6f7f696-khq4q\" (UID: \"261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"
Oct 30 03:15:09.425669 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.425642    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-config\") pod \"korrel8r-66d6f7f696-khq4q\" (UID: \"261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"
Oct 30 03:15:09.428008 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.427986    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-tls\") pod \"korrel8r-66d6f7f696-khq4q\" (UID: \"261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"
Oct 30 03:15:09.445778 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.445755    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtfrm\" (UniqueName: \"kubernetes.io/projected/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-kube-api-access-wtfrm\") pod \"korrel8r-66d6f7f696-khq4q\" (UID: \"261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"
Oct 30 03:15:09.541120 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.541053    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"
Oct 30 03:15:09.837154 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.837124    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"]
Oct 30 03:15:09.838012 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:15:09.837986    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261b8eeb_b6d4_4fa9_b3ac_e028bfbdaa3f.slice/crio-9f9f4a5638aa9cbfe99853d0401db128913957fc288aaba8b50ca3ce58fec4b5 WatchSource:0}: Error finding container 9f9f4a5638aa9cbfe99853d0401db128913957fc288aaba8b50ca3ce58fec4b5: Status 404 returned error can't find the container with id 9f9f4a5638aa9cbfe99853d0401db128913957fc288aaba8b50ca3ce58fec4b5
Oct 30 03:15:09.928228 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.928169    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q" event={"ID":"261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f","Type":"ContainerStarted","Data":"6baa333d7fe02cbb1320c7b45d4283b03ab971ef4f79f0e36f397f9a432f409e"}
Oct 30 03:15:09.928605 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.928240    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q" event={"ID":"261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f","Type":"ContainerStarted","Data":"9f9f4a5638aa9cbfe99853d0401db128913957fc288aaba8b50ca3ce58fec4b5"}
Oct 30 03:15:09.928605 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.928317    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"
Oct 30 03:15:09.969722 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:09.969676    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q" podStartSLOduration=0.969662812 podStartE2EDuration="969.662812ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:09.968718538 +0000 UTC m=+1990.617766960" watchObservedRunningTime="2026-10-30 03:15:09.969662812 +0000 UTC m=+1990.618711232"
Oct 30 03:15:10.951617 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:10.951588    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:11.071680 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:11.071646    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:11.193226 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:11.193164    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:12.153886 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:12.153851    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:12.357099 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:12.357064    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:15.276840 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:15.276806    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:15.277338 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:15.277059    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="198cf335-0852-48ff-870a-d7ba6113edc9" containerName="otc-container" containerID="cri-o://db4d5072ad24a96f17637b4a82c37e6964d94750d95b32965ce517ec453aa996" gracePeriod=30
Oct 30 03:15:17.632735 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.632712    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:17.691203 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.691125    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-data-science-collector-ta-client-cert\") pod \"198cf335-0852-48ff-870a-d7ba6113edc9\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") "
Oct 30 03:15:17.691203 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.691165    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-otlp-tls-certs\") pod \"198cf335-0852-48ff-870a-d7ba6113edc9\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") "
Oct 30 03:15:17.691401 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.691264    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198cf335-0852-48ff-870a-d7ba6113edc9-kube-api-access-qn7ts\" (UniqueName: \"kubernetes.io/projected/198cf335-0852-48ff-870a-d7ba6113edc9-kube-api-access-qn7ts\") pod \"198cf335-0852-48ff-870a-d7ba6113edc9\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") "
Oct 30 03:15:17.691401 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.691330    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-tls-certs\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-tls-certs\") pod \"198cf335-0852-48ff-870a-d7ba6113edc9\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") "
Oct 30 03:15:17.691691 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.691655    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-monitoring-tls-certs\") pod \"198cf335-0852-48ff-870a-d7ba6113edc9\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") "
Oct 30 03:15:17.691796 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.691738    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/198cf335-0852-48ff-870a-d7ba6113edc9-otc-internal\" (UniqueName: \"kubernetes.io/configmap/198cf335-0852-48ff-870a-d7ba6113edc9-otc-internal\") pod \"198cf335-0852-48ff-870a-d7ba6113edc9\" (UID: \"198cf335-0852-48ff-870a-d7ba6113edc9\") "
Oct 30 03:15:17.692244 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.692197    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/198cf335-0852-48ff-870a-d7ba6113edc9-otc-internal" pod "198cf335-0852-48ff-870a-d7ba6113edc9" (UID: "198cf335-0852-48ff-870a-d7ba6113edc9"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:17.694063 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.694038    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-data-science-collector-ta-client-cert" pod "198cf335-0852-48ff-870a-d7ba6113edc9" (UID: "198cf335-0852-48ff-870a-d7ba6113edc9"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:17.694288 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.694270    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198cf335-0852-48ff-870a-d7ba6113edc9-kube-api-access-qn7ts" pod "198cf335-0852-48ff-870a-d7ba6113edc9" (UID: "198cf335-0852-48ff-870a-d7ba6113edc9"). InnerVolumeSpecName "kube-api-access-qn7ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:17.694461 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.694439    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-monitoring-tls-certs" pod "198cf335-0852-48ff-870a-d7ba6113edc9" (UID: "198cf335-0852-48ff-870a-d7ba6113edc9"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:17.694538 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.694464    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-otlp-tls-certs" pod "198cf335-0852-48ff-870a-d7ba6113edc9" (UID: "198cf335-0852-48ff-870a-d7ba6113edc9"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:17.694708 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.694676    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-tls-certs" pod "198cf335-0852-48ff-870a-d7ba6113edc9" (UID: "198cf335-0852-48ff-870a-d7ba6113edc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:17.792955 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.792931    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn7ts\" (UniqueName: \"kubernetes.io/projected/198cf335-0852-48ff-870a-d7ba6113edc9-kube-api-access-qn7ts\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:17.792955 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.792953    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:17.793088 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.792962    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:17.793088 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.792970    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/198cf335-0852-48ff-870a-d7ba6113edc9-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:17.793088 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.792980    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:17.793088 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.792988    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/198cf335-0852-48ff-870a-d7ba6113edc9-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:17.886610 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.886587    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:17.964953 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.964871    4960 generic.go:356] "Generic (PLEG): container finished" podID="198cf335-0852-48ff-870a-d7ba6113edc9" containerID="db4d5072ad24a96f17637b4a82c37e6964d94750d95b32965ce517ec453aa996" exitCode=0
Oct 30 03:15:17.964953 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.964918    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:17.965134 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.964958    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"198cf335-0852-48ff-870a-d7ba6113edc9","Type":"ContainerDied","Data":"db4d5072ad24a96f17637b4a82c37e6964d94750d95b32965ce517ec453aa996"}
Oct 30 03:15:17.965134 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.965008    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"198cf335-0852-48ff-870a-d7ba6113edc9","Type":"ContainerDied","Data":"74ec6f9994f86fbbd804bc6d7f2eaf156bd0303d75defaaa52194b5246074d81"}
Oct 30 03:15:17.965134 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.965031    4960 scope.go:122] "RemoveContainer" containerID="db4d5072ad24a96f17637b4a82c37e6964d94750d95b32965ce517ec453aa996"
Oct 30 03:15:17.974370 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.974351    4960 scope.go:122] "RemoveContainer" containerID="db4d5072ad24a96f17637b4a82c37e6964d94750d95b32965ce517ec453aa996"
Oct 30 03:15:17.974605 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:15:17.974587    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4d5072ad24a96f17637b4a82c37e6964d94750d95b32965ce517ec453aa996\": container with ID starting with db4d5072ad24a96f17637b4a82c37e6964d94750d95b32965ce517ec453aa996 not found: ID does not exist" containerID="db4d5072ad24a96f17637b4a82c37e6964d94750d95b32965ce517ec453aa996"
Oct 30 03:15:17.974656 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:17.974613    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4d5072ad24a96f17637b4a82c37e6964d94750d95b32965ce517ec453aa996"} err="failed to get container status \"db4d5072ad24a96f17637b4a82c37e6964d94750d95b32965ce517ec453aa996\": rpc error: code = NotFound desc = could not find container \"db4d5072ad24a96f17637b4a82c37e6964d94750d95b32965ce517ec453aa996\": container with ID starting with db4d5072ad24a96f17637b4a82c37e6964d94750d95b32965ce517ec453aa996 not found: ID does not exist"
Oct 30 03:15:18.019567 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.019543    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:18.024919 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.024896    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:18.059999 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.059975    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:18.060342 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.060330    4960 cpu_manager.go:414] "containerMap: removing container" podUID="198cf335-0852-48ff-870a-d7ba6113edc9" containerName="otc-container"
Oct 30 03:15:18.060392 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.060344    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198cf335-0852-48ff-870a-d7ba6113edc9" containerName="otc-container"
Oct 30 03:15:18.060434 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.060425    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="198cf335-0852-48ff-870a-d7ba6113edc9" containerName="otc-container"
Oct 30 03:15:18.064661 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.064645    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.082038 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.082015    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:15:18.082895 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.082870    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\""
Oct 30 03:15:18.083024 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.082910    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:15:18.083024 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.082939    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:15:18.083024 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.082952    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:15:18.083024 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.082959    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:15:18.085504 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.085483    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:18.094344 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.094322    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.094424 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.094361    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfnwd\" (UniqueName: \"kubernetes.io/projected/cf266376-4ee3-470d-935c-aa383b907126-kube-api-access-pfnwd\") pod \"data-science-collector-collector-0\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.094498 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.094480    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cf266376-4ee3-470d-935c-aa383b907126-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.094541 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.094516    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.094575 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.094540    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.094640 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.094628    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.195240 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.195215    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.195328 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.195249    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfnwd\" (UniqueName: \"kubernetes.io/projected/cf266376-4ee3-470d-935c-aa383b907126-kube-api-access-pfnwd\") pod \"data-science-collector-collector-0\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.195328 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.195287    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cf266376-4ee3-470d-935c-aa383b907126-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.195328 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.195309    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.195456 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.195332    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.195456 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.195370    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.196036 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.196013    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cf266376-4ee3-470d-935c-aa383b907126-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.197808 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.197788    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.197885 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.197822    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.197990 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.197971    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.198023 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.197969    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.215639 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.215575    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfnwd\" (UniqueName: \"kubernetes.io/projected/cf266376-4ee3-470d-935c-aa383b907126-kube-api-access-pfnwd\") pod \"data-science-collector-collector-0\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.262321 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.262299    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"]
Oct 30 03:15:18.262534 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.262510    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q" podUID="261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f" containerName="korrel8r" containerID="cri-o://6baa333d7fe02cbb1320c7b45d4283b03ab971ef4f79f0e36f397f9a432f409e" gracePeriod=30
Oct 30 03:15:18.264112 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.264095    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"
Oct 30 03:15:18.378289 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.378263    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:18.542607 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.542584    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"
Oct 30 03:15:18.599785 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.599760    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-kube-api-access-wtfrm\" (UniqueName: \"kubernetes.io/projected/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-kube-api-access-wtfrm\") pod \"261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f\" (UID: \"261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f\") "
Oct 30 03:15:18.599901 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.599813    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-tls\" (UniqueName: \"kubernetes.io/secret/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-tls\") pod \"261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f\" (UID: \"261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f\") "
Oct 30 03:15:18.599972 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.599942    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-config\" (UniqueName: \"kubernetes.io/configmap/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-config\") pod \"261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f\" (UID: \"261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f\") "
Oct 30 03:15:18.600365 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.600338    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-config" pod "261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f" (UID: "261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:18.602017 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.601990    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-tls" pod "261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f" (UID: "261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:18.602106 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.602091    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-kube-api-access-wtfrm" pod "261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f" (UID: "261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f"). InnerVolumeSpecName "kube-api-access-wtfrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:18.669933 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.669912    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:18.671731 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:15:18.671706    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf266376_4ee3_470d_935c_aa383b907126.slice/crio-d588b47fd01bbdff3d0b5424bd91cc6105ae13e9e113dff41e3d0fdbc1e61d01 WatchSource:0}: Error finding container d588b47fd01bbdff3d0b5424bd91cc6105ae13e9e113dff41e3d0fdbc1e61d01: Status 404 returned error can't find the container with id d588b47fd01bbdff3d0b5424bd91cc6105ae13e9e113dff41e3d0fdbc1e61d01
Oct 30 03:15:18.673850 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.673819    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:15:18.673977 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.673871    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:15:18.701074 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.701051    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtfrm\" (UniqueName: \"kubernetes.io/projected/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-kube-api-access-wtfrm\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:18.701074 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.701075    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:18.701275 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.701084    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:18.902533 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.902504    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:18.972340 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.972309    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cf266376-4ee3-470d-935c-aa383b907126","Type":"ContainerStarted","Data":"e624c64b3d77f421ea5f2a634c896a4a3b81c7f0a241ca0427cd321cab3b999c"}
Oct 30 03:15:18.972519 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.972361    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cf266376-4ee3-470d-935c-aa383b907126","Type":"ContainerStarted","Data":"d588b47fd01bbdff3d0b5424bd91cc6105ae13e9e113dff41e3d0fdbc1e61d01"}
Oct 30 03:15:18.973916 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.973885    4960 generic.go:356] "Generic (PLEG): container finished" podID="261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f" containerID="6baa333d7fe02cbb1320c7b45d4283b03ab971ef4f79f0e36f397f9a432f409e" exitCode=0
Oct 30 03:15:18.974042 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.973921    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"
Oct 30 03:15:18.974042 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.973946    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q" event={"ID":"261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f","Type":"ContainerDied","Data":"6baa333d7fe02cbb1320c7b45d4283b03ab971ef4f79f0e36f397f9a432f409e"}
Oct 30 03:15:18.974042 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.973995    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q" event={"ID":"261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f","Type":"ContainerDied","Data":"9f9f4a5638aa9cbfe99853d0401db128913957fc288aaba8b50ca3ce58fec4b5"}
Oct 30 03:15:18.974155 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.974049    4960 scope.go:122] "RemoveContainer" containerID="6baa333d7fe02cbb1320c7b45d4283b03ab971ef4f79f0e36f397f9a432f409e"
Oct 30 03:15:18.982903 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.982887    4960 scope.go:122] "RemoveContainer" containerID="6baa333d7fe02cbb1320c7b45d4283b03ab971ef4f79f0e36f397f9a432f409e"
Oct 30 03:15:18.983164 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:15:18.983147    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6baa333d7fe02cbb1320c7b45d4283b03ab971ef4f79f0e36f397f9a432f409e\": container with ID starting with 6baa333d7fe02cbb1320c7b45d4283b03ab971ef4f79f0e36f397f9a432f409e not found: ID does not exist" containerID="6baa333d7fe02cbb1320c7b45d4283b03ab971ef4f79f0e36f397f9a432f409e"
Oct 30 03:15:18.983263 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:18.983172    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6baa333d7fe02cbb1320c7b45d4283b03ab971ef4f79f0e36f397f9a432f409e"} err="failed to get container status \"6baa333d7fe02cbb1320c7b45d4283b03ab971ef4f79f0e36f397f9a432f409e\": rpc error: code = NotFound desc = could not find container \"6baa333d7fe02cbb1320c7b45d4283b03ab971ef4f79f0e36f397f9a432f409e\": container with ID starting with 6baa333d7fe02cbb1320c7b45d4283b03ab971ef4f79f0e36f397f9a432f409e not found: ID does not exist"
Oct 30 03:15:19.019855 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:19.019815    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.019803357 podStartE2EDuration="1.019803357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:19.018883234 +0000 UTC m=+1999.667931655" watchObservedRunningTime="2026-10-30 03:15:19.019803357 +0000 UTC m=+1999.668851850"
Oct 30 03:15:19.058472 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:19.058438    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"]
Oct 30 03:15:19.059622 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:19.059595    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-khq4q"]
Oct 30 03:15:19.502540 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:19.502506    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="198cf335-0852-48ff-870a-d7ba6113edc9" path="/var/lib/kubelet/pods/198cf335-0852-48ff-870a-d7ba6113edc9/volumes"
Oct 30 03:15:19.503004 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:19.502981    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f" path="/var/lib/kubelet/pods/261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f/volumes"
Oct 30 03:15:20.524932 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:20.524898    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:20.704031 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:20.703990    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:21.555034 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:21.554994    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:22.355215 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.355170    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:22.355462 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.355389    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="cf266376-4ee3-470d-935c-aa383b907126" containerName="otc-container" containerID="cri-o://e624c64b3d77f421ea5f2a634c896a4a3b81c7f0a241ca0427cd321cab3b999c" gracePeriod=30
Oct 30 03:15:22.611085 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.611028    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:22.635071 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.635043    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-data-science-collector-ta-client-cert\") pod \"cf266376-4ee3-470d-935c-aa383b907126\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") "
Oct 30 03:15:22.635215 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.635084    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf266376-4ee3-470d-935c-aa383b907126-otc-internal\" (UniqueName: \"kubernetes.io/configmap/cf266376-4ee3-470d-935c-aa383b907126-otc-internal\") pod \"cf266376-4ee3-470d-935c-aa383b907126\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") "
Oct 30 03:15:22.635277 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.635210    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-otlp-tls-certs\") pod \"cf266376-4ee3-470d-935c-aa383b907126\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") "
Oct 30 03:15:22.635277 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.635257    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-tls-certs\") pod \"cf266376-4ee3-470d-935c-aa383b907126\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") "
Oct 30 03:15:22.635368 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.635297    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf266376-4ee3-470d-935c-aa383b907126-kube-api-access-pfnwd\" (UniqueName: \"kubernetes.io/projected/cf266376-4ee3-470d-935c-aa383b907126-kube-api-access-pfnwd\") pod \"cf266376-4ee3-470d-935c-aa383b907126\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") "
Oct 30 03:15:22.635368 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.635341    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-monitoring-tls-certs\") pod \"cf266376-4ee3-470d-935c-aa383b907126\" (UID: \"cf266376-4ee3-470d-935c-aa383b907126\") "
Oct 30 03:15:22.635506 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.635480    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf266376-4ee3-470d-935c-aa383b907126-otc-internal" pod "cf266376-4ee3-470d-935c-aa383b907126" (UID: "cf266376-4ee3-470d-935c-aa383b907126"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:22.635652 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.635630    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cf266376-4ee3-470d-935c-aa383b907126-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.637525 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.637501    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-monitoring-tls-certs" pod "cf266376-4ee3-470d-935c-aa383b907126" (UID: "cf266376-4ee3-470d-935c-aa383b907126"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:22.637603 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.637527    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-otlp-tls-certs" pod "cf266376-4ee3-470d-935c-aa383b907126" (UID: "cf266376-4ee3-470d-935c-aa383b907126"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:22.637603 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.637549    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-tls-certs" pod "cf266376-4ee3-470d-935c-aa383b907126" (UID: "cf266376-4ee3-470d-935c-aa383b907126"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:22.637932 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.637916    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-data-science-collector-ta-client-cert" pod "cf266376-4ee3-470d-935c-aa383b907126" (UID: "cf266376-4ee3-470d-935c-aa383b907126"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:22.638001 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.637941    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf266376-4ee3-470d-935c-aa383b907126-kube-api-access-pfnwd" pod "cf266376-4ee3-470d-935c-aa383b907126" (UID: "cf266376-4ee3-470d-935c-aa383b907126"). InnerVolumeSpecName "kube-api-access-pfnwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:22.736601 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.736571    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.742243 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.736892    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.742243 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.736920    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfnwd\" (UniqueName: \"kubernetes.io/projected/cf266376-4ee3-470d-935c-aa383b907126-kube-api-access-pfnwd\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.742243 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.736947    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.742243 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.736966    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cf266376-4ee3-470d-935c-aa383b907126-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.994328 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.994254    4960 generic.go:356] "Generic (PLEG): container finished" podID="cf266376-4ee3-470d-935c-aa383b907126" containerID="e624c64b3d77f421ea5f2a634c896a4a3b81c7f0a241ca0427cd321cab3b999c" exitCode=0
Oct 30 03:15:22.994328 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.994303    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:22.994490 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.994339    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cf266376-4ee3-470d-935c-aa383b907126","Type":"ContainerDied","Data":"e624c64b3d77f421ea5f2a634c896a4a3b81c7f0a241ca0427cd321cab3b999c"}
Oct 30 03:15:22.994490 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.994418    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cf266376-4ee3-470d-935c-aa383b907126","Type":"ContainerDied","Data":"d588b47fd01bbdff3d0b5424bd91cc6105ae13e9e113dff41e3d0fdbc1e61d01"}
Oct 30 03:15:22.994490 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:22.994442    4960 scope.go:122] "RemoveContainer" containerID="e624c64b3d77f421ea5f2a634c896a4a3b81c7f0a241ca0427cd321cab3b999c"
Oct 30 03:15:23.004022 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.004006    4960 scope.go:122] "RemoveContainer" containerID="e624c64b3d77f421ea5f2a634c896a4a3b81c7f0a241ca0427cd321cab3b999c"
Oct 30 03:15:23.004326 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:15:23.004303    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e624c64b3d77f421ea5f2a634c896a4a3b81c7f0a241ca0427cd321cab3b999c\": container with ID starting with e624c64b3d77f421ea5f2a634c896a4a3b81c7f0a241ca0427cd321cab3b999c not found: ID does not exist" containerID="e624c64b3d77f421ea5f2a634c896a4a3b81c7f0a241ca0427cd321cab3b999c"
Oct 30 03:15:23.004368 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.004338    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e624c64b3d77f421ea5f2a634c896a4a3b81c7f0a241ca0427cd321cab3b999c"} err="failed to get container status \"e624c64b3d77f421ea5f2a634c896a4a3b81c7f0a241ca0427cd321cab3b999c\": rpc error: code = NotFound desc = could not find container \"e624c64b3d77f421ea5f2a634c896a4a3b81c7f0a241ca0427cd321cab3b999c\": container with ID starting with e624c64b3d77f421ea5f2a634c896a4a3b81c7f0a241ca0427cd321cab3b999c not found: ID does not exist"
Oct 30 03:15:23.048889 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.048860    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:23.053399 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.053376    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:23.091795 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.091772    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:23.092121 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.092110    4960 cpu_manager.go:414] "containerMap: removing container" podUID="cf266376-4ee3-470d-935c-aa383b907126" containerName="otc-container"
Oct 30 03:15:23.092169 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.092122    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf266376-4ee3-470d-935c-aa383b907126" containerName="otc-container"
Oct 30 03:15:23.092169 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.092143    4960 cpu_manager.go:414] "containerMap: removing container" podUID="261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f" containerName="korrel8r"
Oct 30 03:15:23.092169 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.092149    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f" containerName="korrel8r"
Oct 30 03:15:23.092289 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.092237    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf266376-4ee3-470d-935c-aa383b907126" containerName="otc-container"
Oct 30 03:15:23.092289 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.092254    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="261b8eeb-b6d4-4fa9-b3ac-e028bfbdaa3f" containerName="korrel8r"
Oct 30 03:15:23.096328 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.096311    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.116783 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.116761    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:15:23.120278 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.120249    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:15:23.120371 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.120334    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:15:23.120437 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.120411    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:15:23.120544 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.120452    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:15:23.120544 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.120521    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:15:23.121382 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.121364    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:23.140630 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.140600    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.140734 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.140644    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/792b8998-5ac8-4e08-b521-3b8f7f912a9f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.140734 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.140722    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.140831 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.140756    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.140874 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.140861    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hskpr\" (UniqueName: \"kubernetes.io/projected/792b8998-5ac8-4e08-b521-3b8f7f912a9f-kube-api-access-hskpr\") pod \"data-science-collector-collector-0\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.140935 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.140921    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.241941 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.241919    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.242041 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.241958    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.242041 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.241981    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/792b8998-5ac8-4e08-b521-3b8f7f912a9f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.242041 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.242011    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.242235 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.242120    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.242235 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.242167    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hskpr\" (UniqueName: \"kubernetes.io/projected/792b8998-5ac8-4e08-b521-3b8f7f912a9f-kube-api-access-hskpr\") pod \"data-science-collector-collector-0\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.242749 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.242695    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/792b8998-5ac8-4e08-b521-3b8f7f912a9f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.244849 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.244798    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.244849 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.244807    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.245072 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.245057    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.245127 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.245111    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.262966 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.262942    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hskpr\" (UniqueName: \"kubernetes.io/projected/792b8998-5ac8-4e08-b521-3b8f7f912a9f-kube-api-access-hskpr\") pod \"data-science-collector-collector-0\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.408262 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.408240    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:23.502217 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.502122    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf266376-4ee3-470d-935c-aa383b907126" path="/var/lib/kubelet/pods/cf266376-4ee3-470d-935c-aa383b907126/volumes"
Oct 30 03:15:23.702521 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.702493    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:23.703074 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:15:23.703049    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792b8998_5ac8_4e08_b521_3b8f7f912a9f.slice/crio-c0d4a5dc09e0d3fa6815e0dc262d1abb3774e4cfec8f3893b26e214568a23740 WatchSource:0}: Error finding container c0d4a5dc09e0d3fa6815e0dc262d1abb3774e4cfec8f3893b26e214568a23740: Status 404 returned error can't find the container with id c0d4a5dc09e0d3fa6815e0dc262d1abb3774e4cfec8f3893b26e214568a23740
Oct 30 03:15:23.705261 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.705232    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:15:23.705352 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:23.705286    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:15:23.955949 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:15:23.955914    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:15:24.001837 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:24.001804    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"792b8998-5ac8-4e08-b521-3b8f7f912a9f","Type":"ContainerStarted","Data":"9b451711e86c1f8bfa20956bc8ad032838a75a663b61d8982503cf141d9615e9"}
Oct 30 03:15:24.001975 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:24.001856    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"792b8998-5ac8-4e08-b521-3b8f7f912a9f","Type":"ContainerStarted","Data":"c0d4a5dc09e0d3fa6815e0dc262d1abb3774e4cfec8f3893b26e214568a23740"}
Oct 30 03:15:24.048442 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:24.048350    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.048336957 podStartE2EDuration="1.048336957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:24.04702077 +0000 UTC m=+2004.696069194" watchObservedRunningTime="2026-10-30 03:15:24.048336957 +0000 UTC m=+2004.697385377"
Oct 30 03:15:27.439087 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:27.439053    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:27.553768 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:27.553730    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:27.665312 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:27.665277    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:27.779413 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:27.779346    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:27.840526 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:27.840495    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"]
Oct 30 03:15:27.844253 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:27.844232    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"
Oct 30 03:15:27.862145 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:27.862118    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"]
Oct 30 03:15:27.876226 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:27.876202    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/48d786a1-20d2-40f3-b737-7db03cdc0ce7-tls\") pod \"korrel8r-66d6f7f696-c7xvj\" (UID: \"48d786a1-20d2-40f3-b737-7db03cdc0ce7\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"
Oct 30 03:15:27.876324 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:27.876237    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/48d786a1-20d2-40f3-b737-7db03cdc0ce7-config\") pod \"korrel8r-66d6f7f696-c7xvj\" (UID: \"48d786a1-20d2-40f3-b737-7db03cdc0ce7\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"
Oct 30 03:15:27.876365 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:27.876319    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf5px\" (UniqueName: \"kubernetes.io/projected/48d786a1-20d2-40f3-b737-7db03cdc0ce7-kube-api-access-qf5px\") pod \"korrel8r-66d6f7f696-c7xvj\" (UID: \"48d786a1-20d2-40f3-b737-7db03cdc0ce7\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"
Oct 30 03:15:27.977283 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:27.977246    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf5px\" (UniqueName: \"kubernetes.io/projected/48d786a1-20d2-40f3-b737-7db03cdc0ce7-kube-api-access-qf5px\") pod \"korrel8r-66d6f7f696-c7xvj\" (UID: \"48d786a1-20d2-40f3-b737-7db03cdc0ce7\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"
Oct 30 03:15:27.977464 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:27.977319    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/48d786a1-20d2-40f3-b737-7db03cdc0ce7-tls\") pod \"korrel8r-66d6f7f696-c7xvj\" (UID: \"48d786a1-20d2-40f3-b737-7db03cdc0ce7\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"
Oct 30 03:15:27.977464 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:27.977343    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/48d786a1-20d2-40f3-b737-7db03cdc0ce7-config\") pod \"korrel8r-66d6f7f696-c7xvj\" (UID: \"48d786a1-20d2-40f3-b737-7db03cdc0ce7\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"
Oct 30 03:15:27.978095 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:27.978027    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/48d786a1-20d2-40f3-b737-7db03cdc0ce7-config\") pod \"korrel8r-66d6f7f696-c7xvj\" (UID: \"48d786a1-20d2-40f3-b737-7db03cdc0ce7\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"
Oct 30 03:15:27.980009 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:27.979980    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/48d786a1-20d2-40f3-b737-7db03cdc0ce7-tls\") pod \"korrel8r-66d6f7f696-c7xvj\" (UID: \"48d786a1-20d2-40f3-b737-7db03cdc0ce7\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"
Oct 30 03:15:27.998928 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:27.998892    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf5px\" (UniqueName: \"kubernetes.io/projected/48d786a1-20d2-40f3-b737-7db03cdc0ce7-kube-api-access-qf5px\") pod \"korrel8r-66d6f7f696-c7xvj\" (UID: \"48d786a1-20d2-40f3-b737-7db03cdc0ce7\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"
Oct 30 03:15:28.157507 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:28.157472    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"
Oct 30 03:15:28.447652 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:28.447629    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"]
Oct 30 03:15:28.449447 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:15:28.449414    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d786a1_20d2_40f3_b737_7db03cdc0ce7.slice/crio-837b3b714f2cfc00d681b35baedbb185acd68fcc3defd170ee8d560ac8fff483 WatchSource:0}: Error finding container 837b3b714f2cfc00d681b35baedbb185acd68fcc3defd170ee8d560ac8fff483: Status 404 returned error can't find the container with id 837b3b714f2cfc00d681b35baedbb185acd68fcc3defd170ee8d560ac8fff483
Oct 30 03:15:29.028837 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:29.028803    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj" event={"ID":"48d786a1-20d2-40f3-b737-7db03cdc0ce7","Type":"ContainerStarted","Data":"f9ba8e56cf4af79c057ac71040172a3ed11616d276bbf4c6af39815112536f2d"}
Oct 30 03:15:29.028993 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:29.028858    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj" event={"ID":"48d786a1-20d2-40f3-b737-7db03cdc0ce7","Type":"ContainerStarted","Data":"837b3b714f2cfc00d681b35baedbb185acd68fcc3defd170ee8d560ac8fff483"}
Oct 30 03:15:29.028993 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:29.028886    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"
Oct 30 03:15:29.070050 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:29.070007    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj" podStartSLOduration=2.069993816 podStartE2EDuration="2.069993816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:29.069828907 +0000 UTC m=+2009.718877329" watchObservedRunningTime="2026-10-30 03:15:29.069993816 +0000 UTC m=+2009.719042239"
Oct 30 03:15:29.465573 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:29.465542    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:29.584524 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:29.584490    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:30.792627 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:30.792587    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:30.962979 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:30.962942    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:31.361934 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.361902    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:31.362104 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.362083    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="792b8998-5ac8-4e08-b521-3b8f7f912a9f" containerName="otc-container" containerID="cri-o://9b451711e86c1f8bfa20956bc8ad032838a75a663b61d8982503cf141d9615e9" gracePeriod=30
Oct 30 03:15:31.620815 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.620760    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:31.709094 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.709064    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-monitoring-tls-certs\") pod \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") "
Oct 30 03:15:31.709244 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.709111    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/792b8998-5ac8-4e08-b521-3b8f7f912a9f-otc-internal\" (UniqueName: \"kubernetes.io/configmap/792b8998-5ac8-4e08-b521-3b8f7f912a9f-otc-internal\") pod \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") "
Oct 30 03:15:31.709244 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.709134    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-otlp-tls-certs\") pod \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") "
Oct 30 03:15:31.709244 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.709167    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-data-science-collector-ta-client-cert\") pod \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") "
Oct 30 03:15:31.709244 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.709204    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/792b8998-5ac8-4e08-b521-3b8f7f912a9f-kube-api-access-hskpr\" (UniqueName: \"kubernetes.io/projected/792b8998-5ac8-4e08-b521-3b8f7f912a9f-kube-api-access-hskpr\") pod \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") "
Oct 30 03:15:31.709445 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.709262    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-tls-certs\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-tls-certs\") pod \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\" (UID: \"792b8998-5ac8-4e08-b521-3b8f7f912a9f\") "
Oct 30 03:15:31.709541 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.709516    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792b8998-5ac8-4e08-b521-3b8f7f912a9f-otc-internal" pod "792b8998-5ac8-4e08-b521-3b8f7f912a9f" (UID: "792b8998-5ac8-4e08-b521-3b8f7f912a9f"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:31.711618 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.711587    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-otlp-tls-certs" pod "792b8998-5ac8-4e08-b521-3b8f7f912a9f" (UID: "792b8998-5ac8-4e08-b521-3b8f7f912a9f"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:31.711618 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.711608    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-monitoring-tls-certs" pod "792b8998-5ac8-4e08-b521-3b8f7f912a9f" (UID: "792b8998-5ac8-4e08-b521-3b8f7f912a9f"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:31.711751 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.711625    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-tls-certs" pod "792b8998-5ac8-4e08-b521-3b8f7f912a9f" (UID: "792b8998-5ac8-4e08-b521-3b8f7f912a9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:31.711883 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.711857    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-data-science-collector-ta-client-cert" pod "792b8998-5ac8-4e08-b521-3b8f7f912a9f" (UID: "792b8998-5ac8-4e08-b521-3b8f7f912a9f"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:31.711883 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.711865    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792b8998-5ac8-4e08-b521-3b8f7f912a9f-kube-api-access-hskpr" pod "792b8998-5ac8-4e08-b521-3b8f7f912a9f" (UID: "792b8998-5ac8-4e08-b521-3b8f7f912a9f"). InnerVolumeSpecName "kube-api-access-hskpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:31.809911 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.809883    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:31.809911 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.809909    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hskpr\" (UniqueName: \"kubernetes.io/projected/792b8998-5ac8-4e08-b521-3b8f7f912a9f-kube-api-access-hskpr\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:31.810292 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.809919    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:31.810292 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.809929    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:31.810292 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.809937    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/792b8998-5ac8-4e08-b521-3b8f7f912a9f-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:31.810292 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:31.809945    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/792b8998-5ac8-4e08-b521-3b8f7f912a9f-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:32.044635 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.044563    4960 generic.go:356] "Generic (PLEG): container finished" podID="792b8998-5ac8-4e08-b521-3b8f7f912a9f" containerID="9b451711e86c1f8bfa20956bc8ad032838a75a663b61d8982503cf141d9615e9" exitCode=0
Oct 30 03:15:32.044752 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.044630    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.044752 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.044654    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"792b8998-5ac8-4e08-b521-3b8f7f912a9f","Type":"ContainerDied","Data":"9b451711e86c1f8bfa20956bc8ad032838a75a663b61d8982503cf141d9615e9"}
Oct 30 03:15:32.044752 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.044730    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"792b8998-5ac8-4e08-b521-3b8f7f912a9f","Type":"ContainerDied","Data":"c0d4a5dc09e0d3fa6815e0dc262d1abb3774e4cfec8f3893b26e214568a23740"}
Oct 30 03:15:32.044848 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.044755    4960 scope.go:122] "RemoveContainer" containerID="9b451711e86c1f8bfa20956bc8ad032838a75a663b61d8982503cf141d9615e9"
Oct 30 03:15:32.054314 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.054296    4960 scope.go:122] "RemoveContainer" containerID="9b451711e86c1f8bfa20956bc8ad032838a75a663b61d8982503cf141d9615e9"
Oct 30 03:15:32.054585 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:15:32.054566    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b451711e86c1f8bfa20956bc8ad032838a75a663b61d8982503cf141d9615e9\": container with ID starting with 9b451711e86c1f8bfa20956bc8ad032838a75a663b61d8982503cf141d9615e9 not found: ID does not exist" containerID="9b451711e86c1f8bfa20956bc8ad032838a75a663b61d8982503cf141d9615e9"
Oct 30 03:15:32.054647 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.054590    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b451711e86c1f8bfa20956bc8ad032838a75a663b61d8982503cf141d9615e9"} err="failed to get container status \"9b451711e86c1f8bfa20956bc8ad032838a75a663b61d8982503cf141d9615e9\": rpc error: code = NotFound desc = could not find container \"9b451711e86c1f8bfa20956bc8ad032838a75a663b61d8982503cf141d9615e9\": container with ID starting with 9b451711e86c1f8bfa20956bc8ad032838a75a663b61d8982503cf141d9615e9 not found: ID does not exist"
Oct 30 03:15:32.100541 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.100518    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:32.114052 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.114030    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:32.139924 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.139903    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:32.140281 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.140269    4960 cpu_manager.go:414] "containerMap: removing container" podUID="792b8998-5ac8-4e08-b521-3b8f7f912a9f" containerName="otc-container"
Oct 30 03:15:32.140329 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.140283    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="792b8998-5ac8-4e08-b521-3b8f7f912a9f" containerName="otc-container"
Oct 30 03:15:32.140364 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.140351    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="792b8998-5ac8-4e08-b521-3b8f7f912a9f" containerName="otc-container"
Oct 30 03:15:32.144796 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.144782    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.165268 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.165247    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:32.165991 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.165968    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:15:32.165991 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.165967    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:15:32.166157 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.166045    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:15:32.166157 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.166095    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:15:32.166157 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.166124    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:15:32.181615 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.181598    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\""
Oct 30 03:15:32.213537 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.213518    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.213625 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.213565    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.213625 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.213587    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.213714 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.213674    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.213714 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.213708    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.213807 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.213729    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nn4j\" (UniqueName: \"kubernetes.io/projected/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-kube-api-access-9nn4j\") pod \"data-science-collector-collector-0\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.314444 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.314396    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.314444 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.314428    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.314569 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.314451    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.314569 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.314467    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.314569 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.314490    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nn4j\" (UniqueName: \"kubernetes.io/projected/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-kube-api-access-9nn4j\") pod \"data-science-collector-collector-0\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.314569 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.314542    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.315489 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.315464    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.317168 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.317147    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.317264 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.317143    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.317264 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.317223    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.317387 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.317370    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.334881 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.334859    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nn4j\" (UniqueName: \"kubernetes.io/projected/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-kube-api-access-9nn4j\") pod \"data-science-collector-collector-0\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.457228 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.457174    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:32.751701 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.751680    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:32.753173 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:15:32.753147    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c62165_d6f9_4d10_b8a0_a2bffb1ee529.slice/crio-1f6291e6ae0bcc2f379bbdd6ee9ef7cdfd164f3557cc84892251fdd24dfc51b0 WatchSource:0}: Error finding container 1f6291e6ae0bcc2f379bbdd6ee9ef7cdfd164f3557cc84892251fdd24dfc51b0: Status 404 returned error can't find the container with id 1f6291e6ae0bcc2f379bbdd6ee9ef7cdfd164f3557cc84892251fdd24dfc51b0
Oct 30 03:15:32.755480 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.755445    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:15:32.755566 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:32.755506    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:15:33.051841 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:33.051762    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529","Type":"ContainerStarted","Data":"2986e99516b0eea8068d9c29235d57dd2364ac7163f2ee52e4c2e7aa054666e4"}
Oct 30 03:15:33.051841 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:33.051826    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529","Type":"ContainerStarted","Data":"1f6291e6ae0bcc2f379bbdd6ee9ef7cdfd164f3557cc84892251fdd24dfc51b0"}
Oct 30 03:15:33.099019 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:33.098976    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.098961807 podStartE2EDuration="1.098961807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:33.09613519 +0000 UTC m=+2013.745183610" watchObservedRunningTime="2026-10-30 03:15:33.098961807 +0000 UTC m=+2013.748010228"
Oct 30 03:15:33.504122 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:33.504088    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="792b8998-5ac8-4e08-b521-3b8f7f912a9f" path="/var/lib/kubelet/pods/792b8998-5ac8-4e08-b521-3b8f7f912a9f/volumes"
Oct 30 03:15:36.398595 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:36.398563    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:36.602414 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:36.602381    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:36.737574 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:36.737496    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:36.759172 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:36.759139    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"]
Oct 30 03:15:36.759330 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:36.759310    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj" podUID="48d786a1-20d2-40f3-b737-7db03cdc0ce7" containerName="korrel8r" containerID="cri-o://f9ba8e56cf4af79c057ac71040172a3ed11616d276bbf4c6af39815112536f2d" gracePeriod=30
Oct 30 03:15:36.761579 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:36.761559    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"
Oct 30 03:15:37.014373 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.014346    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"
Oct 30 03:15:37.054100 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.054072    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48d786a1-20d2-40f3-b737-7db03cdc0ce7-tls\" (UniqueName: \"kubernetes.io/secret/48d786a1-20d2-40f3-b737-7db03cdc0ce7-tls\") pod \"48d786a1-20d2-40f3-b737-7db03cdc0ce7\" (UID: \"48d786a1-20d2-40f3-b737-7db03cdc0ce7\") "
Oct 30 03:15:37.054247 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.054136    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48d786a1-20d2-40f3-b737-7db03cdc0ce7-config\" (UniqueName: \"kubernetes.io/configmap/48d786a1-20d2-40f3-b737-7db03cdc0ce7-config\") pod \"48d786a1-20d2-40f3-b737-7db03cdc0ce7\" (UID: \"48d786a1-20d2-40f3-b737-7db03cdc0ce7\") "
Oct 30 03:15:37.054247 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.054214    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48d786a1-20d2-40f3-b737-7db03cdc0ce7-kube-api-access-qf5px\" (UniqueName: \"kubernetes.io/projected/48d786a1-20d2-40f3-b737-7db03cdc0ce7-kube-api-access-qf5px\") pod \"48d786a1-20d2-40f3-b737-7db03cdc0ce7\" (UID: \"48d786a1-20d2-40f3-b737-7db03cdc0ce7\") "
Oct 30 03:15:37.054523 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.054493    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d786a1-20d2-40f3-b737-7db03cdc0ce7-config" pod "48d786a1-20d2-40f3-b737-7db03cdc0ce7" (UID: "48d786a1-20d2-40f3-b737-7db03cdc0ce7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:37.054632 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.054577    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/48d786a1-20d2-40f3-b737-7db03cdc0ce7-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:37.057710 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.057665    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48d786a1-20d2-40f3-b737-7db03cdc0ce7-tls" pod "48d786a1-20d2-40f3-b737-7db03cdc0ce7" (UID: "48d786a1-20d2-40f3-b737-7db03cdc0ce7"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:37.057833 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.057726    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d786a1-20d2-40f3-b737-7db03cdc0ce7-kube-api-access-qf5px" pod "48d786a1-20d2-40f3-b737-7db03cdc0ce7" (UID: "48d786a1-20d2-40f3-b737-7db03cdc0ce7"). InnerVolumeSpecName "kube-api-access-qf5px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:37.072263 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.072230    4960 generic.go:356] "Generic (PLEG): container finished" podID="48d786a1-20d2-40f3-b737-7db03cdc0ce7" containerID="f9ba8e56cf4af79c057ac71040172a3ed11616d276bbf4c6af39815112536f2d" exitCode=0
Oct 30 03:15:37.072374 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.072284    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"
Oct 30 03:15:37.072374 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.072308    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj" event={"ID":"48d786a1-20d2-40f3-b737-7db03cdc0ce7","Type":"ContainerDied","Data":"f9ba8e56cf4af79c057ac71040172a3ed11616d276bbf4c6af39815112536f2d"}
Oct 30 03:15:37.072487 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.072381    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj" event={"ID":"48d786a1-20d2-40f3-b737-7db03cdc0ce7","Type":"ContainerDied","Data":"837b3b714f2cfc00d681b35baedbb185acd68fcc3defd170ee8d560ac8fff483"}
Oct 30 03:15:37.072487 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.072405    4960 scope.go:122] "RemoveContainer" containerID="f9ba8e56cf4af79c057ac71040172a3ed11616d276bbf4c6af39815112536f2d"
Oct 30 03:15:37.082572 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.082551    4960 scope.go:122] "RemoveContainer" containerID="f9ba8e56cf4af79c057ac71040172a3ed11616d276bbf4c6af39815112536f2d"
Oct 30 03:15:37.082863 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:15:37.082846    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ba8e56cf4af79c057ac71040172a3ed11616d276bbf4c6af39815112536f2d\": container with ID starting with f9ba8e56cf4af79c057ac71040172a3ed11616d276bbf4c6af39815112536f2d not found: ID does not exist" containerID="f9ba8e56cf4af79c057ac71040172a3ed11616d276bbf4c6af39815112536f2d"
Oct 30 03:15:37.082929 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.082871    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ba8e56cf4af79c057ac71040172a3ed11616d276bbf4c6af39815112536f2d"} err="failed to get container status \"f9ba8e56cf4af79c057ac71040172a3ed11616d276bbf4c6af39815112536f2d\": rpc error: code = NotFound desc = could not find container \"f9ba8e56cf4af79c057ac71040172a3ed11616d276bbf4c6af39815112536f2d\": container with ID starting with f9ba8e56cf4af79c057ac71040172a3ed11616d276bbf4c6af39815112536f2d not found: ID does not exist"
Oct 30 03:15:37.130390 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.130370    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"]
Oct 30 03:15:37.134324 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.134299    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-c7xvj"]
Oct 30 03:15:37.155248 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.155224    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/48d786a1-20d2-40f3-b737-7db03cdc0ce7-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:37.155248 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.155246    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf5px\" (UniqueName: \"kubernetes.io/projected/48d786a1-20d2-40f3-b737-7db03cdc0ce7-kube-api-access-qf5px\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:37.503742 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:37.503713    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48d786a1-20d2-40f3-b737-7db03cdc0ce7" path="/var/lib/kubelet/pods/48d786a1-20d2-40f3-b737-7db03cdc0ce7/volumes"
Oct 30 03:15:38.315554 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:38.315498    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:38.446012 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:38.445978    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:38.646479 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:38.646386    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:39.292868 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:39.292833    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:40.426845 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.426806    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:40.427316 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.427026    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="d8c62165-d6f9-4d10-b8a0-a2bffb1ee529" containerName="otc-container" containerID="cri-o://2986e99516b0eea8068d9c29235d57dd2364ac7163f2ee52e4c2e7aa054666e4" gracePeriod=30
Oct 30 03:15:40.675959 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.675931    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:40.784477 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.784382    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-kube-api-access-9nn4j\" (UniqueName: \"kubernetes.io/projected/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-kube-api-access-9nn4j\") pod \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") "
Oct 30 03:15:40.784477 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.784420    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-otlp-tls-certs\") pod \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") "
Oct 30 03:15:40.784688 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.784484    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-data-science-collector-ta-client-cert\") pod \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") "
Oct 30 03:15:40.784688 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.784543    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-monitoring-tls-certs\") pod \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") "
Oct 30 03:15:40.784688 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.784585    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-tls-certs\") pod \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") "
Oct 30 03:15:40.784688 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.784645    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-otc-internal\") pod \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\" (UID: \"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529\") "
Oct 30 03:15:40.785108 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.785074    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-otc-internal" pod "d8c62165-d6f9-4d10-b8a0-a2bffb1ee529" (UID: "d8c62165-d6f9-4d10-b8a0-a2bffb1ee529"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:40.786842 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.786813    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-kube-api-access-9nn4j" pod "d8c62165-d6f9-4d10-b8a0-a2bffb1ee529" (UID: "d8c62165-d6f9-4d10-b8a0-a2bffb1ee529"). InnerVolumeSpecName "kube-api-access-9nn4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:40.787264 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.787239    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-otlp-tls-certs" pod "d8c62165-d6f9-4d10-b8a0-a2bffb1ee529" (UID: "d8c62165-d6f9-4d10-b8a0-a2bffb1ee529"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:40.787264 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.787251    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-monitoring-tls-certs" pod "d8c62165-d6f9-4d10-b8a0-a2bffb1ee529" (UID: "d8c62165-d6f9-4d10-b8a0-a2bffb1ee529"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:40.787405 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.787384    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-tls-certs" pod "d8c62165-d6f9-4d10-b8a0-a2bffb1ee529" (UID: "d8c62165-d6f9-4d10-b8a0-a2bffb1ee529"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:40.787455 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.787407    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-data-science-collector-ta-client-cert" pod "d8c62165-d6f9-4d10-b8a0-a2bffb1ee529" (UID: "d8c62165-d6f9-4d10-b8a0-a2bffb1ee529"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:40.885908 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.885884    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:40.885908 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.885906    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nn4j\" (UniqueName: \"kubernetes.io/projected/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-kube-api-access-9nn4j\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:40.886040 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.885916    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:40.886040 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.885926    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:40.886040 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.885935    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:40.886040 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:40.885943    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:41.093089 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.093057    4960 generic.go:356] "Generic (PLEG): container finished" podID="d8c62165-d6f9-4d10-b8a0-a2bffb1ee529" containerID="2986e99516b0eea8068d9c29235d57dd2364ac7163f2ee52e4c2e7aa054666e4" exitCode=0
Oct 30 03:15:41.093202 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.093110    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.093202 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.093124    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529","Type":"ContainerDied","Data":"2986e99516b0eea8068d9c29235d57dd2364ac7163f2ee52e4c2e7aa054666e4"}
Oct 30 03:15:41.093289 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.093206    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d8c62165-d6f9-4d10-b8a0-a2bffb1ee529","Type":"ContainerDied","Data":"1f6291e6ae0bcc2f379bbdd6ee9ef7cdfd164f3557cc84892251fdd24dfc51b0"}
Oct 30 03:15:41.093289 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.093239    4960 scope.go:122] "RemoveContainer" containerID="2986e99516b0eea8068d9c29235d57dd2364ac7163f2ee52e4c2e7aa054666e4"
Oct 30 03:15:41.102367 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.102269    4960 scope.go:122] "RemoveContainer" containerID="2986e99516b0eea8068d9c29235d57dd2364ac7163f2ee52e4c2e7aa054666e4"
Oct 30 03:15:41.102501 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:15:41.102485    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2986e99516b0eea8068d9c29235d57dd2364ac7163f2ee52e4c2e7aa054666e4\": container with ID starting with 2986e99516b0eea8068d9c29235d57dd2364ac7163f2ee52e4c2e7aa054666e4 not found: ID does not exist" containerID="2986e99516b0eea8068d9c29235d57dd2364ac7163f2ee52e4c2e7aa054666e4"
Oct 30 03:15:41.102550 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.102508    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2986e99516b0eea8068d9c29235d57dd2364ac7163f2ee52e4c2e7aa054666e4"} err="failed to get container status \"2986e99516b0eea8068d9c29235d57dd2364ac7163f2ee52e4c2e7aa054666e4\": rpc error: code = NotFound desc = could not find container \"2986e99516b0eea8068d9c29235d57dd2364ac7163f2ee52e4c2e7aa054666e4\": container with ID starting with 2986e99516b0eea8068d9c29235d57dd2364ac7163f2ee52e4c2e7aa054666e4 not found: ID does not exist"
Oct 30 03:15:41.146820 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.146799    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:41.160582 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.160558    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:41.187823 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.187802    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:41.188159 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.188148    4960 cpu_manager.go:414] "containerMap: removing container" podUID="48d786a1-20d2-40f3-b737-7db03cdc0ce7" containerName="korrel8r"
Oct 30 03:15:41.188218 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.188161    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d786a1-20d2-40f3-b737-7db03cdc0ce7" containerName="korrel8r"
Oct 30 03:15:41.188218 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.188170    4960 cpu_manager.go:414] "containerMap: removing container" podUID="d8c62165-d6f9-4d10-b8a0-a2bffb1ee529" containerName="otc-container"
Oct 30 03:15:41.188218 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.188175    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c62165-d6f9-4d10-b8a0-a2bffb1ee529" containerName="otc-container"
Oct 30 03:15:41.188313 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.188277    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d786a1-20d2-40f3-b737-7db03cdc0ce7" containerName="korrel8r"
Oct 30 03:15:41.188313 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.188285    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c62165-d6f9-4d10-b8a0-a2bffb1ee529" containerName="otc-container"
Oct 30 03:15:41.192450 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.192435    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.211059 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.211035    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:15:41.211201 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.211126    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:15:41.211201 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.211160    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:15:41.211319 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.211238    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:15:41.211319 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.211277    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:15:41.220719 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.214513    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:41.224099 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.224075    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:15:41.288640 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.288612    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.288737 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.288663    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.288786 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.288768    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.288822 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.288793    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-226xz\" (UniqueName: \"kubernetes.io/projected/547decca-1a2f-4088-bde7-0c1b7bb55d81-kube-api-access-226xz\") pod \"data-science-collector-collector-0\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.288872 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.288831    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.288946 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.288927    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/547decca-1a2f-4088-bde7-0c1b7bb55d81-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.389652 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.389601    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/547decca-1a2f-4088-bde7-0c1b7bb55d81-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.389652 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.389644    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.389762 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.389672    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.389762 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.389709    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.389762 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.389730    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-226xz\" (UniqueName: \"kubernetes.io/projected/547decca-1a2f-4088-bde7-0c1b7bb55d81-kube-api-access-226xz\") pod \"data-science-collector-collector-0\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.389861 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.389765    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.390392 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.390364    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/547decca-1a2f-4088-bde7-0c1b7bb55d81-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.392355 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.392334    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.392455 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.392436    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.392704 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.392685    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.392799 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.392777    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.410569 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.410551    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-226xz\" (UniqueName: \"kubernetes.io/projected/547decca-1a2f-4088-bde7-0c1b7bb55d81-kube-api-access-226xz\") pod \"data-science-collector-collector-0\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.502677 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.502637    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c62165-d6f9-4d10-b8a0-a2bffb1ee529" path="/var/lib/kubelet/pods/d8c62165-d6f9-4d10-b8a0-a2bffb1ee529/volumes"
Oct 30 03:15:41.504731 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.504706    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:41.840399 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.840373    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:41.841376 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:15:41.841349    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547decca_1a2f_4088_bde7_0c1b7bb55d81.slice/crio-ba1116362b5f173640b87c0f88a1f3c8096ce9637a7a6001a2fb892bdfbef1ce WatchSource:0}: Error finding container ba1116362b5f173640b87c0f88a1f3c8096ce9637a7a6001a2fb892bdfbef1ce: Status 404 returned error can't find the container with id ba1116362b5f173640b87c0f88a1f3c8096ce9637a7a6001a2fb892bdfbef1ce
Oct 30 03:15:41.843637 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.843606    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:15:41.843745 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:41.843675    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:15:42.100960 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:42.100877    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"547decca-1a2f-4088-bde7-0c1b7bb55d81","Type":"ContainerStarted","Data":"a2592059657777122c3a115d1cbcae91af7fc8d87dad9f7bdf027751c7b87673"}
Oct 30 03:15:42.100960 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:42.100940    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"547decca-1a2f-4088-bde7-0c1b7bb55d81","Type":"ContainerStarted","Data":"ba1116362b5f173640b87c0f88a1f3c8096ce9637a7a6001a2fb892bdfbef1ce"}
Oct 30 03:15:42.148941 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:42.148895    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.148880143 podStartE2EDuration="1.148880143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:42.147857533 +0000 UTC m=+2022.796905958" watchObservedRunningTime="2026-10-30 03:15:42.148880143 +0000 UTC m=+2022.797928564"
Oct 30 03:15:44.745616 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:44.745547    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:44.858955 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:44.858911    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:45.059890 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:45.059807    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:45.173561 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:45.173530    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:46.663991 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:46.663942    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:46.791626 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:46.791595    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:46.995952 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:46.995852    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:48.502864 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.502831    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:48.503366 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.503043    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="547decca-1a2f-4088-bde7-0c1b7bb55d81" containerName="otc-container" containerID="cri-o://a2592059657777122c3a115d1cbcae91af7fc8d87dad9f7bdf027751c7b87673" gracePeriod=30
Oct 30 03:15:48.754100 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.754047    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:48.845355 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.845329    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547decca-1a2f-4088-bde7-0c1b7bb55d81-kube-api-access-226xz\" (UniqueName: \"kubernetes.io/projected/547decca-1a2f-4088-bde7-0c1b7bb55d81-kube-api-access-226xz\") pod \"547decca-1a2f-4088-bde7-0c1b7bb55d81\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") "
Oct 30 03:15:48.845487 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.845427    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-data-science-collector-ta-client-cert\") pod \"547decca-1a2f-4088-bde7-0c1b7bb55d81\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") "
Oct 30 03:15:48.845487 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.845460    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-tls-certs\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-tls-certs\") pod \"547decca-1a2f-4088-bde7-0c1b7bb55d81\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") "
Oct 30 03:15:48.845487 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.845478    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/547decca-1a2f-4088-bde7-0c1b7bb55d81-otc-internal\" (UniqueName: \"kubernetes.io/configmap/547decca-1a2f-4088-bde7-0c1b7bb55d81-otc-internal\") pod \"547decca-1a2f-4088-bde7-0c1b7bb55d81\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") "
Oct 30 03:15:48.845647 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.845499    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-otlp-tls-certs\") pod \"547decca-1a2f-4088-bde7-0c1b7bb55d81\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") "
Oct 30 03:15:48.845647 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.845528    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-monitoring-tls-certs\") pod \"547decca-1a2f-4088-bde7-0c1b7bb55d81\" (UID: \"547decca-1a2f-4088-bde7-0c1b7bb55d81\") "
Oct 30 03:15:48.845915 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.845886    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547decca-1a2f-4088-bde7-0c1b7bb55d81-otc-internal" pod "547decca-1a2f-4088-bde7-0c1b7bb55d81" (UID: "547decca-1a2f-4088-bde7-0c1b7bb55d81"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:48.847765 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.847744    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547decca-1a2f-4088-bde7-0c1b7bb55d81-kube-api-access-226xz" pod "547decca-1a2f-4088-bde7-0c1b7bb55d81" (UID: "547decca-1a2f-4088-bde7-0c1b7bb55d81"). InnerVolumeSpecName "kube-api-access-226xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:48.848170 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.848147    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-data-science-collector-ta-client-cert" pod "547decca-1a2f-4088-bde7-0c1b7bb55d81" (UID: "547decca-1a2f-4088-bde7-0c1b7bb55d81"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:48.848291 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.848193    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-otlp-tls-certs" pod "547decca-1a2f-4088-bde7-0c1b7bb55d81" (UID: "547decca-1a2f-4088-bde7-0c1b7bb55d81"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:48.848335 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.848298    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-monitoring-tls-certs" pod "547decca-1a2f-4088-bde7-0c1b7bb55d81" (UID: "547decca-1a2f-4088-bde7-0c1b7bb55d81"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:48.848335 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.848310    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-tls-certs" pod "547decca-1a2f-4088-bde7-0c1b7bb55d81" (UID: "547decca-1a2f-4088-bde7-0c1b7bb55d81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:48.946151 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.946120    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:48.946151 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.946150    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/547decca-1a2f-4088-bde7-0c1b7bb55d81-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:48.946303 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.946161    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:48.946303 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.946169    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:48.946303 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.946202    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-226xz\" (UniqueName: \"kubernetes.io/projected/547decca-1a2f-4088-bde7-0c1b7bb55d81-kube-api-access-226xz\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:48.946303 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:48.946219    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/547decca-1a2f-4088-bde7-0c1b7bb55d81-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:49.135026 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.134998    4960 generic.go:356] "Generic (PLEG): container finished" podID="547decca-1a2f-4088-bde7-0c1b7bb55d81" containerID="a2592059657777122c3a115d1cbcae91af7fc8d87dad9f7bdf027751c7b87673" exitCode=0
Oct 30 03:15:49.135155 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.135045    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.135155 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.135090    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"547decca-1a2f-4088-bde7-0c1b7bb55d81","Type":"ContainerDied","Data":"a2592059657777122c3a115d1cbcae91af7fc8d87dad9f7bdf027751c7b87673"}
Oct 30 03:15:49.135155 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.135138    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"547decca-1a2f-4088-bde7-0c1b7bb55d81","Type":"ContainerDied","Data":"ba1116362b5f173640b87c0f88a1f3c8096ce9637a7a6001a2fb892bdfbef1ce"}
Oct 30 03:15:49.135295 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.135162    4960 scope.go:122] "RemoveContainer" containerID="a2592059657777122c3a115d1cbcae91af7fc8d87dad9f7bdf027751c7b87673"
Oct 30 03:15:49.144044 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.144028    4960 scope.go:122] "RemoveContainer" containerID="a2592059657777122c3a115d1cbcae91af7fc8d87dad9f7bdf027751c7b87673"
Oct 30 03:15:49.144314 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:15:49.144295    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2592059657777122c3a115d1cbcae91af7fc8d87dad9f7bdf027751c7b87673\": container with ID starting with a2592059657777122c3a115d1cbcae91af7fc8d87dad9f7bdf027751c7b87673 not found: ID does not exist" containerID="a2592059657777122c3a115d1cbcae91af7fc8d87dad9f7bdf027751c7b87673"
Oct 30 03:15:49.144364 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.144322    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2592059657777122c3a115d1cbcae91af7fc8d87dad9f7bdf027751c7b87673"} err="failed to get container status \"a2592059657777122c3a115d1cbcae91af7fc8d87dad9f7bdf027751c7b87673\": rpc error: code = NotFound desc = could not find container \"a2592059657777122c3a115d1cbcae91af7fc8d87dad9f7bdf027751c7b87673\": container with ID starting with a2592059657777122c3a115d1cbcae91af7fc8d87dad9f7bdf027751c7b87673 not found: ID does not exist"
Oct 30 03:15:49.191448 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.191423    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:49.204886 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.204867    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:49.229643 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.229621    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:49.229971 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.229959    4960 cpu_manager.go:414] "containerMap: removing container" podUID="547decca-1a2f-4088-bde7-0c1b7bb55d81" containerName="otc-container"
Oct 30 03:15:49.230015 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.229972    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="547decca-1a2f-4088-bde7-0c1b7bb55d81" containerName="otc-container"
Oct 30 03:15:49.230060 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.230051    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="547decca-1a2f-4088-bde7-0c1b7bb55d81" containerName="otc-container"
Oct 30 03:15:49.234406 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.234390    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.254894 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.254850    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:15:49.255038 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.254958    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:15:49.255103 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.255086    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:15:49.255161 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.255119    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-770491e3\""
Oct 30 03:15:49.256615 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.256593    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:49.267261 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.267238    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:15:49.267261 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.267241    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:15:49.349387 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.349362    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/05188274-708e-4e68-a4c8-e08747c27165-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.349387 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.349388    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8l7g\" (UniqueName: \"kubernetes.io/projected/05188274-708e-4e68-a4c8-e08747c27165-kube-api-access-b8l7g\") pod \"data-science-collector-collector-0\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.349563 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.349409    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.349563 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.349491    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.349563 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.349546    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.349702 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.349578    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.450024 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.449968    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/05188274-708e-4e68-a4c8-e08747c27165-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.450024 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.450005    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8l7g\" (UniqueName: \"kubernetes.io/projected/05188274-708e-4e68-a4c8-e08747c27165-kube-api-access-b8l7g\") pod \"data-science-collector-collector-0\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.450227 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.450029    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.450227 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.450070    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.450227 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.450091    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.450227 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.450106    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.450953 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.450857    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/05188274-708e-4e68-a4c8-e08747c27165-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.452608 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.452589    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.452722 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.452694    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.452794 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.452742    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.452905 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.452887    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.471188 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.471156    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8l7g\" (UniqueName: \"kubernetes.io/projected/05188274-708e-4e68-a4c8-e08747c27165-kube-api-access-b8l7g\") pod \"data-science-collector-collector-0\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.502421 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.502397    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="547decca-1a2f-4088-bde7-0c1b7bb55d81" path="/var/lib/kubelet/pods/547decca-1a2f-4088-bde7-0c1b7bb55d81/volumes"
Oct 30 03:15:49.547280 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.547253    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:49.839730 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.839706    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:49.841023 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:15:49.840992    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05188274_708e_4e68_a4c8_e08747c27165.slice/crio-3ff0af654a25722824749d1593083d6c49995b98ff42c64f41655101173f4a40 WatchSource:0}: Error finding container 3ff0af654a25722824749d1593083d6c49995b98ff42c64f41655101173f4a40: Status 404 returned error can't find the container with id 3ff0af654a25722824749d1593083d6c49995b98ff42c64f41655101173f4a40
Oct 30 03:15:49.843394 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.843356    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:15:49.843497 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:49.843430    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:15:50.141677 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:50.141590    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"05188274-708e-4e68-a4c8-e08747c27165","Type":"ContainerStarted","Data":"e2ce6ff769ab5bf394501de7656483c9ff40429ce618e4267884f67eea378a5b"}
Oct 30 03:15:50.141677 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:50.141656    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"05188274-708e-4e68-a4c8-e08747c27165","Type":"ContainerStarted","Data":"3ff0af654a25722824749d1593083d6c49995b98ff42c64f41655101173f4a40"}
Oct 30 03:15:50.185871 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:50.185818    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.185804742 podStartE2EDuration="1.185804742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:50.1844199 +0000 UTC m=+2030.833468322" watchObservedRunningTime="2026-10-30 03:15:50.185804742 +0000 UTC m=+2030.834853226"
Oct 30 03:15:52.996583 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:52.996545    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:53.197213 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:53.197165    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:53.306053 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:53.305983    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:53.953760 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:15:53.953730    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:15:54.986747 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:54.986706    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:55.114749 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:55.114711    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:56.519981 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:56.519938    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:56.520370 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:56.520202    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="05188274-708e-4e68-a4c8-e08747c27165" containerName="otc-container" containerID="cri-o://e2ce6ff769ab5bf394501de7656483c9ff40429ce618e4267884f67eea378a5b" gracePeriod=30
Oct 30 03:15:57.774856 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:57.774825    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:57.928207 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:57.928095    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-tls-certs\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-tls-certs\") pod \"05188274-708e-4e68-a4c8-e08747c27165\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") "
Oct 30 03:15:57.928207 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:57.928156    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-data-science-collector-ta-client-cert\") pod \"05188274-708e-4e68-a4c8-e08747c27165\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") "
Oct 30 03:15:57.928207 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:57.928201    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05188274-708e-4e68-a4c8-e08747c27165-otc-internal\" (UniqueName: \"kubernetes.io/configmap/05188274-708e-4e68-a4c8-e08747c27165-otc-internal\") pod \"05188274-708e-4e68-a4c8-e08747c27165\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") "
Oct 30 03:15:57.928483 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:57.928219    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-monitoring-tls-certs\") pod \"05188274-708e-4e68-a4c8-e08747c27165\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") "
Oct 30 03:15:57.928483 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:57.928288    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05188274-708e-4e68-a4c8-e08747c27165-kube-api-access-b8l7g\" (UniqueName: \"kubernetes.io/projected/05188274-708e-4e68-a4c8-e08747c27165-kube-api-access-b8l7g\") pod \"05188274-708e-4e68-a4c8-e08747c27165\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") "
Oct 30 03:15:57.928483 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:57.928305    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-otlp-tls-certs\") pod \"05188274-708e-4e68-a4c8-e08747c27165\" (UID: \"05188274-708e-4e68-a4c8-e08747c27165\") "
Oct 30 03:15:57.928660 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:57.928631    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05188274-708e-4e68-a4c8-e08747c27165-otc-internal" pod "05188274-708e-4e68-a4c8-e08747c27165" (UID: "05188274-708e-4e68-a4c8-e08747c27165"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:57.930802 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:57.930747    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-data-science-collector-ta-client-cert" pod "05188274-708e-4e68-a4c8-e08747c27165" (UID: "05188274-708e-4e68-a4c8-e08747c27165"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:57.930802 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:57.930749    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05188274-708e-4e68-a4c8-e08747c27165-kube-api-access-b8l7g" pod "05188274-708e-4e68-a4c8-e08747c27165" (UID: "05188274-708e-4e68-a4c8-e08747c27165"). InnerVolumeSpecName "kube-api-access-b8l7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:57.930963 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:57.930811    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-monitoring-tls-certs" pod "05188274-708e-4e68-a4c8-e08747c27165" (UID: "05188274-708e-4e68-a4c8-e08747c27165"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:57.931058 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:57.931044    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-otlp-tls-certs" pod "05188274-708e-4e68-a4c8-e08747c27165" (UID: "05188274-708e-4e68-a4c8-e08747c27165"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:57.931107 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:57.931068    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-tls-certs" pod "05188274-708e-4e68-a4c8-e08747c27165" (UID: "05188274-708e-4e68-a4c8-e08747c27165"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:58.028855 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.028831    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:58.028855 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.028856    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:58.029010 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.028867    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/05188274-708e-4e68-a4c8-e08747c27165-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:58.029010 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.028882    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:58.029010 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.028896    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8l7g\" (UniqueName: \"kubernetes.io/projected/05188274-708e-4e68-a4c8-e08747c27165-kube-api-access-b8l7g\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:58.029010 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.028904    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/05188274-708e-4e68-a4c8-e08747c27165-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:15:58.183484 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.183407    4960 generic.go:356] "Generic (PLEG): container finished" podID="05188274-708e-4e68-a4c8-e08747c27165" containerID="e2ce6ff769ab5bf394501de7656483c9ff40429ce618e4267884f67eea378a5b" exitCode=0
Oct 30 03:15:58.183484 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.183479    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.183628 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.183498    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"05188274-708e-4e68-a4c8-e08747c27165","Type":"ContainerDied","Data":"e2ce6ff769ab5bf394501de7656483c9ff40429ce618e4267884f67eea378a5b"}
Oct 30 03:15:58.183628 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.183578    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"05188274-708e-4e68-a4c8-e08747c27165","Type":"ContainerDied","Data":"3ff0af654a25722824749d1593083d6c49995b98ff42c64f41655101173f4a40"}
Oct 30 03:15:58.183628 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.183603    4960 scope.go:122] "RemoveContainer" containerID="e2ce6ff769ab5bf394501de7656483c9ff40429ce618e4267884f67eea378a5b"
Oct 30 03:15:58.192567 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.192546    4960 scope.go:122] "RemoveContainer" containerID="e2ce6ff769ab5bf394501de7656483c9ff40429ce618e4267884f67eea378a5b"
Oct 30 03:15:58.192847 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:15:58.192827    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ce6ff769ab5bf394501de7656483c9ff40429ce618e4267884f67eea378a5b\": container with ID starting with e2ce6ff769ab5bf394501de7656483c9ff40429ce618e4267884f67eea378a5b not found: ID does not exist" containerID="e2ce6ff769ab5bf394501de7656483c9ff40429ce618e4267884f67eea378a5b"
Oct 30 03:15:58.192895 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.192854    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ce6ff769ab5bf394501de7656483c9ff40429ce618e4267884f67eea378a5b"} err="failed to get container status \"e2ce6ff769ab5bf394501de7656483c9ff40429ce618e4267884f67eea378a5b\": rpc error: code = NotFound desc = could not find container \"e2ce6ff769ab5bf394501de7656483c9ff40429ce618e4267884f67eea378a5b\": container with ID starting with e2ce6ff769ab5bf394501de7656483c9ff40429ce618e4267884f67eea378a5b not found: ID does not exist"
Oct 30 03:15:58.237811 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.237782    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:58.241204 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.241169    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:58.275864 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.275844    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:58.276244 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.276232    4960 cpu_manager.go:414] "containerMap: removing container" podUID="05188274-708e-4e68-a4c8-e08747c27165" containerName="otc-container"
Oct 30 03:15:58.276284 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.276246    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05188274-708e-4e68-a4c8-e08747c27165" containerName="otc-container"
Oct 30 03:15:58.276325 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.276311    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="05188274-708e-4e68-a4c8-e08747c27165" containerName="otc-container"
Oct 30 03:15:58.280650 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.280632    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.297486 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.297457    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:15:58.297486 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.297467    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:15:58.297691 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.297547    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:15:58.297691 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.297564    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:15:58.297691 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.297554    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:15:58.297691 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.297597    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:15:58.299052 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.299029    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:58.432908 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.432876    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6c8672cd-6d59-41d4-97fb-2aaa826ff345-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.433013 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.432911    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.433013 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.432941    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.433013 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.433001    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwkzw\" (UniqueName: \"kubernetes.io/projected/6c8672cd-6d59-41d4-97fb-2aaa826ff345-kube-api-access-nwkzw\") pod \"data-science-collector-collector-0\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.433173 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.433153    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.433238 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.433205    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.534077 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.534019    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6c8672cd-6d59-41d4-97fb-2aaa826ff345-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.534077 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.534057    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.534289 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.534089    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.534289 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.534115    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwkzw\" (UniqueName: \"kubernetes.io/projected/6c8672cd-6d59-41d4-97fb-2aaa826ff345-kube-api-access-nwkzw\") pod \"data-science-collector-collector-0\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.534289 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.534225    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.534289 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.534249    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.534871 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.534795    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6c8672cd-6d59-41d4-97fb-2aaa826ff345-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.536762 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.536736    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.536855 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.536786    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.536855 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.536847    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.536930 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.536859    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.555273 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.555253    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwkzw\" (UniqueName: \"kubernetes.io/projected/6c8672cd-6d59-41d4-97fb-2aaa826ff345-kube-api-access-nwkzw\") pod \"data-science-collector-collector-0\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.592525 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.592494    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:15:58.891112 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.891079    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:15:58.892492 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:15:58.892464    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8672cd_6d59_41d4_97fb_2aaa826ff345.slice/crio-4b43235f11fca4157f4088c5be53b82b670adba57e06f68c43ddac16c0900fa3 WatchSource:0}: Error finding container 4b43235f11fca4157f4088c5be53b82b670adba57e06f68c43ddac16c0900fa3: Status 404 returned error can't find the container with id 4b43235f11fca4157f4088c5be53b82b670adba57e06f68c43ddac16c0900fa3
Oct 30 03:15:58.894785 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.894752    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:15:58.894900 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:58.894814    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:15:59.192735 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:59.192662    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6c8672cd-6d59-41d4-97fb-2aaa826ff345","Type":"ContainerStarted","Data":"b470b441d1dfce3b1bb7c94234ffece6e8f96bfc81a9d1104d72e4eeb9eb2ad1"}
Oct 30 03:15:59.192735 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:59.192720    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6c8672cd-6d59-41d4-97fb-2aaa826ff345","Type":"ContainerStarted","Data":"4b43235f11fca4157f4088c5be53b82b670adba57e06f68c43ddac16c0900fa3"}
Oct 30 03:15:59.235852 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:59.235803    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.235791181 podStartE2EDuration="1.235791181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:59.23492444 +0000 UTC m=+2039.883972863" watchObservedRunningTime="2026-10-30 03:15:59.235791181 +0000 UTC m=+2039.884839664"
Oct 30 03:15:59.502494 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:15:59.502424    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05188274-708e-4e68-a4c8-e08747c27165" path="/var/lib/kubelet/pods/05188274-708e-4e68-a4c8-e08747c27165/volumes"
Oct 30 03:16:01.584058 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:01.584023    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:01.796466 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:01.796418    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:01.933004 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:01.932938    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"]
Oct 30 03:16:01.937965 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:01.937946    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"
Oct 30 03:16:01.945049 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:01.945016    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:01.960521 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:01.960496    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"]
Oct 30 03:16:02.065224 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:02.065198    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/02000e11-76d4-4a66-98c7-71badd267c90-config\") pod \"korrel8r-66d6f7f696-m4ftn\" (UID: \"02000e11-76d4-4a66-98c7-71badd267c90\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"
Oct 30 03:16:02.065333 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:02.065230    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/02000e11-76d4-4a66-98c7-71badd267c90-tls\") pod \"korrel8r-66d6f7f696-m4ftn\" (UID: \"02000e11-76d4-4a66-98c7-71badd267c90\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"
Oct 30 03:16:02.065333 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:02.065297    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbx7r\" (UniqueName: \"kubernetes.io/projected/02000e11-76d4-4a66-98c7-71badd267c90-kube-api-access-qbx7r\") pod \"korrel8r-66d6f7f696-m4ftn\" (UID: \"02000e11-76d4-4a66-98c7-71badd267c90\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"
Oct 30 03:16:02.166455 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:02.166426    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/02000e11-76d4-4a66-98c7-71badd267c90-config\") pod \"korrel8r-66d6f7f696-m4ftn\" (UID: \"02000e11-76d4-4a66-98c7-71badd267c90\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"
Oct 30 03:16:02.166590 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:02.166460    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/02000e11-76d4-4a66-98c7-71badd267c90-tls\") pod \"korrel8r-66d6f7f696-m4ftn\" (UID: \"02000e11-76d4-4a66-98c7-71badd267c90\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"
Oct 30 03:16:02.166590 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:02.166480    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbx7r\" (UniqueName: \"kubernetes.io/projected/02000e11-76d4-4a66-98c7-71badd267c90-kube-api-access-qbx7r\") pod \"korrel8r-66d6f7f696-m4ftn\" (UID: \"02000e11-76d4-4a66-98c7-71badd267c90\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"
Oct 30 03:16:02.167173 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:02.167149    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/02000e11-76d4-4a66-98c7-71badd267c90-config\") pod \"korrel8r-66d6f7f696-m4ftn\" (UID: \"02000e11-76d4-4a66-98c7-71badd267c90\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"
Oct 30 03:16:02.169038 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:02.169018    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/02000e11-76d4-4a66-98c7-71badd267c90-tls\") pod \"korrel8r-66d6f7f696-m4ftn\" (UID: \"02000e11-76d4-4a66-98c7-71badd267c90\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"
Oct 30 03:16:02.188217 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:02.188148    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbx7r\" (UniqueName: \"kubernetes.io/projected/02000e11-76d4-4a66-98c7-71badd267c90-kube-api-access-qbx7r\") pod \"korrel8r-66d6f7f696-m4ftn\" (UID: \"02000e11-76d4-4a66-98c7-71badd267c90\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"
Oct 30 03:16:02.250610 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:02.250587    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"
Oct 30 03:16:02.542458 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:02.542435    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"]
Oct 30 03:16:02.544120 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:16:02.544090    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02000e11_76d4_4a66_98c7_71badd267c90.slice/crio-4e500c4bf9a3a3211bad83394f42fa93b72b1ec1d370f8f59ee7153642e366e5 WatchSource:0}: Error finding container 4e500c4bf9a3a3211bad83394f42fa93b72b1ec1d370f8f59ee7153642e366e5: Status 404 returned error can't find the container with id 4e500c4bf9a3a3211bad83394f42fa93b72b1ec1d370f8f59ee7153642e366e5
Oct 30 03:16:03.213465 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:03.213428    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn" event={"ID":"02000e11-76d4-4a66-98c7-71badd267c90","Type":"ContainerStarted","Data":"07b0a3676ced2d0a5ff0027f051746e8b81f30da4618178c626b93720ef1fed7"}
Oct 30 03:16:03.213932 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:03.213492    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn" event={"ID":"02000e11-76d4-4a66-98c7-71badd267c90","Type":"ContainerStarted","Data":"4e500c4bf9a3a3211bad83394f42fa93b72b1ec1d370f8f59ee7153642e366e5"}
Oct 30 03:16:03.213932 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:03.213549    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"
Oct 30 03:16:03.266014 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:03.265970    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn" podStartSLOduration=2.265959132 podStartE2EDuration="2.265959132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:03.265229873 +0000 UTC m=+2043.914278294" watchObservedRunningTime="2026-10-30 03:16:03.265959132 +0000 UTC m=+2043.915007579"
Oct 30 03:16:03.594152 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:03.594124    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:03.719679 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:03.719650    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:03.833303 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:03.833266    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:04.906668 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:04.906638    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:05.135877 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:05.135845    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:05.594052 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:05.594018    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:05.594296 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:05.594268    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="6c8672cd-6d59-41d4-97fb-2aaa826ff345" containerName="otc-container" containerID="cri-o://b470b441d1dfce3b1bb7c94234ffece6e8f96bfc81a9d1104d72e4eeb9eb2ad1" gracePeriod=30
Oct 30 03:16:06.843709 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:06.843682    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.007922 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.007850    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c8672cd-6d59-41d4-97fb-2aaa826ff345-kube-api-access-nwkzw\" (UniqueName: \"kubernetes.io/projected/6c8672cd-6d59-41d4-97fb-2aaa826ff345-kube-api-access-nwkzw\") pod \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") "
Oct 30 03:16:07.007922 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.007889    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-monitoring-tls-certs\") pod \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") "
Oct 30 03:16:07.007922 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.007924    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-tls-certs\") pod \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") "
Oct 30 03:16:07.008173 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.007940    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-otlp-tls-certs\") pod \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") "
Oct 30 03:16:07.008173 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.007980    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c8672cd-6d59-41d4-97fb-2aaa826ff345-otc-internal\" (UniqueName: \"kubernetes.io/configmap/6c8672cd-6d59-41d4-97fb-2aaa826ff345-otc-internal\") pod \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") "
Oct 30 03:16:07.008173 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.008005    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-data-science-collector-ta-client-cert\") pod \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\" (UID: \"6c8672cd-6d59-41d4-97fb-2aaa826ff345\") "
Oct 30 03:16:07.008465 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.008435    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c8672cd-6d59-41d4-97fb-2aaa826ff345-otc-internal" pod "6c8672cd-6d59-41d4-97fb-2aaa826ff345" (UID: "6c8672cd-6d59-41d4-97fb-2aaa826ff345"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:07.010318 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.010294    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8672cd-6d59-41d4-97fb-2aaa826ff345-kube-api-access-nwkzw" pod "6c8672cd-6d59-41d4-97fb-2aaa826ff345" (UID: "6c8672cd-6d59-41d4-97fb-2aaa826ff345"). InnerVolumeSpecName "kube-api-access-nwkzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:07.010548 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.010510    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-monitoring-tls-certs" pod "6c8672cd-6d59-41d4-97fb-2aaa826ff345" (UID: "6c8672cd-6d59-41d4-97fb-2aaa826ff345"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:07.010648 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.010575    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-data-science-collector-ta-client-cert" pod "6c8672cd-6d59-41d4-97fb-2aaa826ff345" (UID: "6c8672cd-6d59-41d4-97fb-2aaa826ff345"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:07.010747 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.010730    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-tls-certs" pod "6c8672cd-6d59-41d4-97fb-2aaa826ff345" (UID: "6c8672cd-6d59-41d4-97fb-2aaa826ff345"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:07.010801 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.010779    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-otlp-tls-certs" pod "6c8672cd-6d59-41d4-97fb-2aaa826ff345" (UID: "6c8672cd-6d59-41d4-97fb-2aaa826ff345"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:07.108677 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.108652    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwkzw\" (UniqueName: \"kubernetes.io/projected/6c8672cd-6d59-41d4-97fb-2aaa826ff345-kube-api-access-nwkzw\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:07.108677 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.108674    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:07.108795 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.108685    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:07.108795 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.108694    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:07.108795 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.108702    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6c8672cd-6d59-41d4-97fb-2aaa826ff345-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:07.108795 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.108711    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6c8672cd-6d59-41d4-97fb-2aaa826ff345-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:07.232663 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.232633    4960 generic.go:356] "Generic (PLEG): container finished" podID="6c8672cd-6d59-41d4-97fb-2aaa826ff345" containerID="b470b441d1dfce3b1bb7c94234ffece6e8f96bfc81a9d1104d72e4eeb9eb2ad1" exitCode=0
Oct 30 03:16:07.232771 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.232686    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.232771 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.232717    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6c8672cd-6d59-41d4-97fb-2aaa826ff345","Type":"ContainerDied","Data":"b470b441d1dfce3b1bb7c94234ffece6e8f96bfc81a9d1104d72e4eeb9eb2ad1"}
Oct 30 03:16:07.232899 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.232782    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6c8672cd-6d59-41d4-97fb-2aaa826ff345","Type":"ContainerDied","Data":"4b43235f11fca4157f4088c5be53b82b670adba57e06f68c43ddac16c0900fa3"}
Oct 30 03:16:07.232899 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.232810    4960 scope.go:122] "RemoveContainer" containerID="b470b441d1dfce3b1bb7c94234ffece6e8f96bfc81a9d1104d72e4eeb9eb2ad1"
Oct 30 03:16:07.242506 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.242486    4960 scope.go:122] "RemoveContainer" containerID="b470b441d1dfce3b1bb7c94234ffece6e8f96bfc81a9d1104d72e4eeb9eb2ad1"
Oct 30 03:16:07.242764 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:16:07.242747    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b470b441d1dfce3b1bb7c94234ffece6e8f96bfc81a9d1104d72e4eeb9eb2ad1\": container with ID starting with b470b441d1dfce3b1bb7c94234ffece6e8f96bfc81a9d1104d72e4eeb9eb2ad1 not found: ID does not exist" containerID="b470b441d1dfce3b1bb7c94234ffece6e8f96bfc81a9d1104d72e4eeb9eb2ad1"
Oct 30 03:16:07.242819 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.242777    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b470b441d1dfce3b1bb7c94234ffece6e8f96bfc81a9d1104d72e4eeb9eb2ad1"} err="failed to get container status \"b470b441d1dfce3b1bb7c94234ffece6e8f96bfc81a9d1104d72e4eeb9eb2ad1\": rpc error: code = NotFound desc = could not find container \"b470b441d1dfce3b1bb7c94234ffece6e8f96bfc81a9d1104d72e4eeb9eb2ad1\": container with ID starting with b470b441d1dfce3b1bb7c94234ffece6e8f96bfc81a9d1104d72e4eeb9eb2ad1 not found: ID does not exist"
Oct 30 03:16:07.287554 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.287502    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:07.290957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.290935    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:07.324581 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.324560    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:07.324909 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.324896    4960 cpu_manager.go:414] "containerMap: removing container" podUID="6c8672cd-6d59-41d4-97fb-2aaa826ff345" containerName="otc-container"
Oct 30 03:16:07.324957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.324909    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c8672cd-6d59-41d4-97fb-2aaa826ff345" containerName="otc-container"
Oct 30 03:16:07.325000 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.324988    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c8672cd-6d59-41d4-97fb-2aaa826ff345" containerName="otc-container"
Oct 30 03:16:07.329356 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.329342    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.345894 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.345869    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-50fd48e3\""
Oct 30 03:16:07.346950 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.346812    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:16:07.346950 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.346869    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:16:07.346950 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.346903    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:16:07.346950 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.346932    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:16:07.347120 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.346959    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:16:07.347745 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.347728    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:07.502460 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.502432    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c8672cd-6d59-41d4-97fb-2aaa826ff345" path="/var/lib/kubelet/pods/6c8672cd-6d59-41d4-97fb-2aaa826ff345/volumes"
Oct 30 03:16:07.511804 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.511784    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.511855 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.511820    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.511914 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.511895    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7qdd\" (UniqueName: \"kubernetes.io/projected/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-kube-api-access-c7qdd\") pod \"data-science-collector-collector-0\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.511980 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.511963    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.512014 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.511991    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.512047 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.512017    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.612385 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.612361    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.612485 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.612396    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.612485 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.612418    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7qdd\" (UniqueName: \"kubernetes.io/projected/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-kube-api-access-c7qdd\") pod \"data-science-collector-collector-0\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.612485 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.612441    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.612485 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.612470    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.612690 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.612504    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.613082 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.613056    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.615149 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.615123    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.615263 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.615196    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.615263 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.615214    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.615333 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.615284    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.633314 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.633296    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7qdd\" (UniqueName: \"kubernetes.io/projected/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-kube-api-access-c7qdd\") pod \"data-science-collector-collector-0\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.641023 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.641007    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:07.933799 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.933771    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:07.934868 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:16:07.934837    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5b487f_2a9b_47ee_b3e4_5691ff30bee6.slice/crio-369a7c33bc1c11c75da970ab8212c11ad53896d63ebb35f6491a5b2fa2ccbd59 WatchSource:0}: Error finding container 369a7c33bc1c11c75da970ab8212c11ad53896d63ebb35f6491a5b2fa2ccbd59: Status 404 returned error can't find the container with id 369a7c33bc1c11c75da970ab8212c11ad53896d63ebb35f6491a5b2fa2ccbd59
Oct 30 03:16:07.937203 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.937144    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:16:07.937281 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:07.937265    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:16:08.240598 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:08.240529    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6","Type":"ContainerStarted","Data":"3e63b3a34b9deb97d618582f074cb7616d08ae8d114cb8fcf846bee6fa2cb6eb"}
Oct 30 03:16:08.240598 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:08.240584    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6","Type":"ContainerStarted","Data":"369a7c33bc1c11c75da970ab8212c11ad53896d63ebb35f6491a5b2fa2ccbd59"}
Oct 30 03:16:08.283038 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:08.282991    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.282978695 podStartE2EDuration="1.282978695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:08.281223226 +0000 UTC m=+2048.930271647" watchObservedRunningTime="2026-10-30 03:16:08.282978695 +0000 UTC m=+2048.932027115"
Oct 30 03:16:11.077153 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:11.077123    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:11.199164 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:11.199128    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:11.381265 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:11.381194    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:11.449090 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:11.449055    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"]
Oct 30 03:16:11.449377 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:11.449327    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn" podUID="02000e11-76d4-4a66-98c7-71badd267c90" containerName="korrel8r" containerID="cri-o://07b0a3676ced2d0a5ff0027f051746e8b81f30da4618178c626b93720ef1fed7" gracePeriod=30
Oct 30 03:16:11.451787 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:11.451766    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"
Oct 30 03:16:11.509600 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:11.509562    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:11.701019 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:11.700994    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"
Oct 30 03:16:11.850076 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:11.850045    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02000e11-76d4-4a66-98c7-71badd267c90-tls\" (UniqueName: \"kubernetes.io/secret/02000e11-76d4-4a66-98c7-71badd267c90-tls\") pod \"02000e11-76d4-4a66-98c7-71badd267c90\" (UID: \"02000e11-76d4-4a66-98c7-71badd267c90\") "
Oct 30 03:16:11.850250 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:11.850152    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02000e11-76d4-4a66-98c7-71badd267c90-kube-api-access-qbx7r\" (UniqueName: \"kubernetes.io/projected/02000e11-76d4-4a66-98c7-71badd267c90-kube-api-access-qbx7r\") pod \"02000e11-76d4-4a66-98c7-71badd267c90\" (UID: \"02000e11-76d4-4a66-98c7-71badd267c90\") "
Oct 30 03:16:11.850250 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:11.850216    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02000e11-76d4-4a66-98c7-71badd267c90-config\" (UniqueName: \"kubernetes.io/configmap/02000e11-76d4-4a66-98c7-71badd267c90-config\") pod \"02000e11-76d4-4a66-98c7-71badd267c90\" (UID: \"02000e11-76d4-4a66-98c7-71badd267c90\") "
Oct 30 03:16:11.850605 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:11.850582    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02000e11-76d4-4a66-98c7-71badd267c90-config" pod "02000e11-76d4-4a66-98c7-71badd267c90" (UID: "02000e11-76d4-4a66-98c7-71badd267c90"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:11.852162 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:11.852138    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02000e11-76d4-4a66-98c7-71badd267c90-tls" pod "02000e11-76d4-4a66-98c7-71badd267c90" (UID: "02000e11-76d4-4a66-98c7-71badd267c90"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:11.852259 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:11.852238    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02000e11-76d4-4a66-98c7-71badd267c90-kube-api-access-qbx7r" pod "02000e11-76d4-4a66-98c7-71badd267c90" (UID: "02000e11-76d4-4a66-98c7-71badd267c90"). InnerVolumeSpecName "kube-api-access-qbx7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:11.951080 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:11.951030    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/02000e11-76d4-4a66-98c7-71badd267c90-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:11.951080 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:11.951055    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbx7r\" (UniqueName: \"kubernetes.io/projected/02000e11-76d4-4a66-98c7-71badd267c90-kube-api-access-qbx7r\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:11.951080 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:11.951065    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/02000e11-76d4-4a66-98c7-71badd267c90-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:12.263037 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:12.262968    4960 generic.go:356] "Generic (PLEG): container finished" podID="02000e11-76d4-4a66-98c7-71badd267c90" containerID="07b0a3676ced2d0a5ff0027f051746e8b81f30da4618178c626b93720ef1fed7" exitCode=0
Oct 30 03:16:12.263037 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:12.263019    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"
Oct 30 03:16:12.263500 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:12.263054    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn" event={"ID":"02000e11-76d4-4a66-98c7-71badd267c90","Type":"ContainerDied","Data":"07b0a3676ced2d0a5ff0027f051746e8b81f30da4618178c626b93720ef1fed7"}
Oct 30 03:16:12.263500 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:12.263105    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn" event={"ID":"02000e11-76d4-4a66-98c7-71badd267c90","Type":"ContainerDied","Data":"4e500c4bf9a3a3211bad83394f42fa93b72b1ec1d370f8f59ee7153642e366e5"}
Oct 30 03:16:12.263500 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:12.263127    4960 scope.go:122] "RemoveContainer" containerID="07b0a3676ced2d0a5ff0027f051746e8b81f30da4618178c626b93720ef1fed7"
Oct 30 03:16:12.272760 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:12.272745    4960 scope.go:122] "RemoveContainer" containerID="07b0a3676ced2d0a5ff0027f051746e8b81f30da4618178c626b93720ef1fed7"
Oct 30 03:16:12.273050 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:16:12.273031    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b0a3676ced2d0a5ff0027f051746e8b81f30da4618178c626b93720ef1fed7\": container with ID starting with 07b0a3676ced2d0a5ff0027f051746e8b81f30da4618178c626b93720ef1fed7 not found: ID does not exist" containerID="07b0a3676ced2d0a5ff0027f051746e8b81f30da4618178c626b93720ef1fed7"
Oct 30 03:16:12.273117 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:12.273062    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b0a3676ced2d0a5ff0027f051746e8b81f30da4618178c626b93720ef1fed7"} err="failed to get container status \"07b0a3676ced2d0a5ff0027f051746e8b81f30da4618178c626b93720ef1fed7\": rpc error: code = NotFound desc = could not find container \"07b0a3676ced2d0a5ff0027f051746e8b81f30da4618178c626b93720ef1fed7\": container with ID starting with 07b0a3676ced2d0a5ff0027f051746e8b81f30da4618178c626b93720ef1fed7 not found: ID does not exist"
Oct 30 03:16:12.319546 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:12.319519    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"]
Oct 30 03:16:12.332967 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:12.332942    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-m4ftn"]
Oct 30 03:16:13.059264 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:13.059236    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:13.190545 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:13.190517    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:13.503325 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:13.503293    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02000e11-76d4-4a66-98c7-71badd267c90" path="/var/lib/kubelet/pods/02000e11-76d4-4a66-98c7-71badd267c90/volumes"
Oct 30 03:16:14.022418 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:14.022387    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:14.623949 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:14.623917    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:14.624482 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:14.624236    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="2a5b487f-2a9b-47ee-b3e4-5691ff30bee6" containerName="otc-container" containerID="cri-o://3e63b3a34b9deb97d618582f074cb7616d08ae8d114cb8fcf846bee6fa2cb6eb" gracePeriod=30
Oct 30 03:16:15.871389 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:15.871362    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:15.883416 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:15.883390    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-otc-internal\") pod \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") "
Oct 30 03:16:15.883507 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:15.883429    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-monitoring-tls-certs\") pod \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") "
Oct 30 03:16:15.883507 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:15.883474    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-otlp-tls-certs\") pod \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") "
Oct 30 03:16:15.883667 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:15.883526    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-tls-certs\") pod \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") "
Oct 30 03:16:15.883667 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:15.883642    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-kube-api-access-c7qdd\" (UniqueName: \"kubernetes.io/projected/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-kube-api-access-c7qdd\") pod \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") "
Oct 30 03:16:15.883779 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:15.883708    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-data-science-collector-ta-client-cert\") pod \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\" (UID: \"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6\") "
Oct 30 03:16:15.883835 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:15.883774    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-otc-internal" pod "2a5b487f-2a9b-47ee-b3e4-5691ff30bee6" (UID: "2a5b487f-2a9b-47ee-b3e4-5691ff30bee6"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:15.883979 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:15.883958    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:15.885789 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:15.885763    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-tls-certs" pod "2a5b487f-2a9b-47ee-b3e4-5691ff30bee6" (UID: "2a5b487f-2a9b-47ee-b3e4-5691ff30bee6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:15.885918 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:15.885902    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-monitoring-tls-certs" pod "2a5b487f-2a9b-47ee-b3e4-5691ff30bee6" (UID: "2a5b487f-2a9b-47ee-b3e4-5691ff30bee6"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:15.886082 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:15.886060    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-otlp-tls-certs" pod "2a5b487f-2a9b-47ee-b3e4-5691ff30bee6" (UID: "2a5b487f-2a9b-47ee-b3e4-5691ff30bee6"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:15.886395 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:15.886367    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-kube-api-access-c7qdd" pod "2a5b487f-2a9b-47ee-b3e4-5691ff30bee6" (UID: "2a5b487f-2a9b-47ee-b3e4-5691ff30bee6"). InnerVolumeSpecName "kube-api-access-c7qdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:15.886479 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:15.886398    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-data-science-collector-ta-client-cert" pod "2a5b487f-2a9b-47ee-b3e4-5691ff30bee6" (UID: "2a5b487f-2a9b-47ee-b3e4-5691ff30bee6"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:15.984507 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:15.984439    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:15.984507 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:15.984484    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7qdd\" (UniqueName: \"kubernetes.io/projected/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-kube-api-access-c7qdd\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:15.984507 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:15.984495    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:15.984507 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:15.984505    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:15.984719 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:15.984513    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:16.284563 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.284483    4960 generic.go:356] "Generic (PLEG): container finished" podID="2a5b487f-2a9b-47ee-b3e4-5691ff30bee6" containerID="3e63b3a34b9deb97d618582f074cb7616d08ae8d114cb8fcf846bee6fa2cb6eb" exitCode=0
Oct 30 03:16:16.284563 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.284547    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.284780 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.284573    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6","Type":"ContainerDied","Data":"3e63b3a34b9deb97d618582f074cb7616d08ae8d114cb8fcf846bee6fa2cb6eb"}
Oct 30 03:16:16.284780 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.284660    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"2a5b487f-2a9b-47ee-b3e4-5691ff30bee6","Type":"ContainerDied","Data":"369a7c33bc1c11c75da970ab8212c11ad53896d63ebb35f6491a5b2fa2ccbd59"}
Oct 30 03:16:16.284780 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.284692    4960 scope.go:122] "RemoveContainer" containerID="3e63b3a34b9deb97d618582f074cb7616d08ae8d114cb8fcf846bee6fa2cb6eb"
Oct 30 03:16:16.293818 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.293798    4960 scope.go:122] "RemoveContainer" containerID="3e63b3a34b9deb97d618582f074cb7616d08ae8d114cb8fcf846bee6fa2cb6eb"
Oct 30 03:16:16.294072 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:16:16.294054    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e63b3a34b9deb97d618582f074cb7616d08ae8d114cb8fcf846bee6fa2cb6eb\": container with ID starting with 3e63b3a34b9deb97d618582f074cb7616d08ae8d114cb8fcf846bee6fa2cb6eb not found: ID does not exist" containerID="3e63b3a34b9deb97d618582f074cb7616d08ae8d114cb8fcf846bee6fa2cb6eb"
Oct 30 03:16:16.294117 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.294081    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e63b3a34b9deb97d618582f074cb7616d08ae8d114cb8fcf846bee6fa2cb6eb"} err="failed to get container status \"3e63b3a34b9deb97d618582f074cb7616d08ae8d114cb8fcf846bee6fa2cb6eb\": rpc error: code = NotFound desc = could not find container \"3e63b3a34b9deb97d618582f074cb7616d08ae8d114cb8fcf846bee6fa2cb6eb\": container with ID starting with 3e63b3a34b9deb97d618582f074cb7616d08ae8d114cb8fcf846bee6fa2cb6eb not found: ID does not exist"
Oct 30 03:16:16.338450 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.338429    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:16.341869 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.341841    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:16.375189 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.375164    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:16.375527 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.375514    4960 cpu_manager.go:414] "containerMap: removing container" podUID="02000e11-76d4-4a66-98c7-71badd267c90" containerName="korrel8r"
Oct 30 03:16:16.375570 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.375530    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02000e11-76d4-4a66-98c7-71badd267c90" containerName="korrel8r"
Oct 30 03:16:16.375570 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.375549    4960 cpu_manager.go:414] "containerMap: removing container" podUID="2a5b487f-2a9b-47ee-b3e4-5691ff30bee6" containerName="otc-container"
Oct 30 03:16:16.375570 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.375555    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5b487f-2a9b-47ee-b3e4-5691ff30bee6" containerName="otc-container"
Oct 30 03:16:16.375663 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.375613    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="02000e11-76d4-4a66-98c7-71badd267c90" containerName="korrel8r"
Oct 30 03:16:16.375663 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.375623    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5b487f-2a9b-47ee-b3e4-5691ff30bee6" containerName="otc-container"
Oct 30 03:16:16.380004 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.379988    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.386869 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.386846    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.386961 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.386906    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75fpn\" (UniqueName: \"kubernetes.io/projected/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-kube-api-access-75fpn\") pod \"data-science-collector-collector-0\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.387027 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.387008    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.387069 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.387036    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.387069 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.387053    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.387141 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.387072    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.396913 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.396892    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:16.397102 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.397083    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:16:16.397171 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.397140    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:16:16.397237 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.397215    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:16:16.397272 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.397235    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:16:16.397318 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.397305    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:16:16.397388 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.397377    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:16:16.487445 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.487423    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.487556 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.487467    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75fpn\" (UniqueName: \"kubernetes.io/projected/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-kube-api-access-75fpn\") pod \"data-science-collector-collector-0\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.487556 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.487509    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.487556 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.487528    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.487556 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.487545    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.487712 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.487570    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.488436 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.488411    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.490195 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.490165    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.490286 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.490213    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.490286 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.490224    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.490362 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.490284    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.508437 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.508417    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75fpn\" (UniqueName: \"kubernetes.io/projected/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-kube-api-access-75fpn\") pod \"data-science-collector-collector-0\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.692968 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.692947    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:16.984810 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.984787    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:16.987361 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:16:16.987323    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a17e5b2_a757_498e_b825_2bd9ee4f7fbe.slice/crio-45b632525eff87af5003fb87d2c3203a41e6ba72f1379800f66cc39c88d916b7 WatchSource:0}: Error finding container 45b632525eff87af5003fb87d2c3203a41e6ba72f1379800f66cc39c88d916b7: Status 404 returned error can't find the container with id 45b632525eff87af5003fb87d2c3203a41e6ba72f1379800f66cc39c88d916b7
Oct 30 03:16:16.989523 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.989489    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:16:16.989643 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:16.989548    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:16:17.293543 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:17.293468    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe","Type":"ContainerStarted","Data":"6e2f1a8a438810f739317791a5cdc48fba84243ae7ce75f5113289f2f0b32ad0"}
Oct 30 03:16:17.293543 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:17.293523    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe","Type":"ContainerStarted","Data":"45b632525eff87af5003fb87d2c3203a41e6ba72f1379800f66cc39c88d916b7"}
Oct 30 03:16:17.338754 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:17.338695    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.338682267 podStartE2EDuration="1.338682267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:17.338457968 +0000 UTC m=+2057.987506390" watchObservedRunningTime="2026-10-30 03:16:17.338682267 +0000 UTC m=+2057.987730688"
Oct 30 03:16:17.502298 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:17.502269    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a5b487f-2a9b-47ee-b3e4-5691ff30bee6" path="/var/lib/kubelet/pods/2a5b487f-2a9b-47ee-b3e4-5691ff30bee6/volumes"
Oct 30 03:16:20.194850 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:20.194816    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:20.317385 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:20.317357    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:20.442625 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:20.442584    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:22.045359 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:22.045331    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:22.171071 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:22.171035    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:23.953920 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:16:23.953888    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:16:24.687436 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:24.687402    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:24.687684 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:24.687657    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="3a17e5b2-a757-498e-b825-2bd9ee4f7fbe" containerName="otc-container" containerID="cri-o://6e2f1a8a438810f739317791a5cdc48fba84243ae7ce75f5113289f2f0b32ad0" gracePeriod=30
Oct 30 03:16:25.937750 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:25.937726    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.065078 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.064997    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-data-science-collector-ta-client-cert\") pod \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") "
Oct 30 03:16:26.065078 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.065036    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-kube-api-access-75fpn\" (UniqueName: \"kubernetes.io/projected/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-kube-api-access-75fpn\") pod \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") "
Oct 30 03:16:26.065335 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.065082    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-otc-internal\" (UniqueName: \"kubernetes.io/configmap/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-otc-internal\") pod \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") "
Oct 30 03:16:26.065335 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.065143    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-otlp-tls-certs\") pod \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") "
Oct 30 03:16:26.065335 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.065171    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-tls-certs\") pod \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") "
Oct 30 03:16:26.065489 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.065334    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-monitoring-tls-certs\") pod \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\" (UID: \"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe\") "
Oct 30 03:16:26.065600 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.065570    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-otc-internal" pod "3a17e5b2-a757-498e-b825-2bd9ee4f7fbe" (UID: "3a17e5b2-a757-498e-b825-2bd9ee4f7fbe"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:26.065703 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.065668    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:26.067502 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.067476    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-data-science-collector-ta-client-cert" pod "3a17e5b2-a757-498e-b825-2bd9ee4f7fbe" (UID: "3a17e5b2-a757-498e-b825-2bd9ee4f7fbe"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:26.067653 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.067612    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-monitoring-tls-certs" pod "3a17e5b2-a757-498e-b825-2bd9ee4f7fbe" (UID: "3a17e5b2-a757-498e-b825-2bd9ee4f7fbe"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:26.067653 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.067625    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-otlp-tls-certs" pod "3a17e5b2-a757-498e-b825-2bd9ee4f7fbe" (UID: "3a17e5b2-a757-498e-b825-2bd9ee4f7fbe"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:26.067881 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.067862    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-tls-certs" pod "3a17e5b2-a757-498e-b825-2bd9ee4f7fbe" (UID: "3a17e5b2-a757-498e-b825-2bd9ee4f7fbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:26.067963 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.067947    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-kube-api-access-75fpn" pod "3a17e5b2-a757-498e-b825-2bd9ee4f7fbe" (UID: "3a17e5b2-a757-498e-b825-2bd9ee4f7fbe"). InnerVolumeSpecName "kube-api-access-75fpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:26.166024 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.166002    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:26.166024 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.166024    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:26.166142 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.166032    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:26.166142 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.166041    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:26.166142 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.166052    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75fpn\" (UniqueName: \"kubernetes.io/projected/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe-kube-api-access-75fpn\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:26.334564 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.334485    4960 generic.go:356] "Generic (PLEG): container finished" podID="3a17e5b2-a757-498e-b825-2bd9ee4f7fbe" containerID="6e2f1a8a438810f739317791a5cdc48fba84243ae7ce75f5113289f2f0b32ad0" exitCode=0
Oct 30 03:16:26.334564 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.334534    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.334757 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.334566    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe","Type":"ContainerDied","Data":"6e2f1a8a438810f739317791a5cdc48fba84243ae7ce75f5113289f2f0b32ad0"}
Oct 30 03:16:26.334757 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.334615    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"3a17e5b2-a757-498e-b825-2bd9ee4f7fbe","Type":"ContainerDied","Data":"45b632525eff87af5003fb87d2c3203a41e6ba72f1379800f66cc39c88d916b7"}
Oct 30 03:16:26.334757 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.334638    4960 scope.go:122] "RemoveContainer" containerID="6e2f1a8a438810f739317791a5cdc48fba84243ae7ce75f5113289f2f0b32ad0"
Oct 30 03:16:26.343866 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.343850    4960 scope.go:122] "RemoveContainer" containerID="6e2f1a8a438810f739317791a5cdc48fba84243ae7ce75f5113289f2f0b32ad0"
Oct 30 03:16:26.344160 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:16:26.344136    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e2f1a8a438810f739317791a5cdc48fba84243ae7ce75f5113289f2f0b32ad0\": container with ID starting with 6e2f1a8a438810f739317791a5cdc48fba84243ae7ce75f5113289f2f0b32ad0 not found: ID does not exist" containerID="6e2f1a8a438810f739317791a5cdc48fba84243ae7ce75f5113289f2f0b32ad0"
Oct 30 03:16:26.344284 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.344167    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e2f1a8a438810f739317791a5cdc48fba84243ae7ce75f5113289f2f0b32ad0"} err="failed to get container status \"6e2f1a8a438810f739317791a5cdc48fba84243ae7ce75f5113289f2f0b32ad0\": rpc error: code = NotFound desc = could not find container \"6e2f1a8a438810f739317791a5cdc48fba84243ae7ce75f5113289f2f0b32ad0\": container with ID starting with 6e2f1a8a438810f739317791a5cdc48fba84243ae7ce75f5113289f2f0b32ad0 not found: ID does not exist"
Oct 30 03:16:26.389662 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.389640    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:26.394160 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.394126    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:26.428039 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.428015    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:26.428419 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.428397    4960 cpu_manager.go:414] "containerMap: removing container" podUID="3a17e5b2-a757-498e-b825-2bd9ee4f7fbe" containerName="otc-container"
Oct 30 03:16:26.428419 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.428412    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a17e5b2-a757-498e-b825-2bd9ee4f7fbe" containerName="otc-container"
Oct 30 03:16:26.428536 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.428482    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a17e5b2-a757-498e-b825-2bd9ee4f7fbe" containerName="otc-container"
Oct 30 03:16:26.432980 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.432963    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.449894 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.449869    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:26.458338 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.458316    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:16:26.478640 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.478622    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:16:26.478830 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.478814    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:16:26.479076 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.479064    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:16:26.479114 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.479078    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:16:26.479114 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.479098    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:16:26.569357 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.569326    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qg9x\" (UniqueName: \"kubernetes.io/projected/0568c967-b6cc-458a-ab00-c056ef82630a-kube-api-access-7qg9x\") pod \"data-science-collector-collector-0\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.569479 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.569362    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.569479 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.569388    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.569479 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.569463    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.569639 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.569531    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0568c967-b6cc-458a-ab00-c056ef82630a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.569639 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.569622    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.677065 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.670780    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.677065 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.670857    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.677065 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.670900    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.677065 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.671006    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0568c967-b6cc-458a-ab00-c056ef82630a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.677065 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.671070    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.677065 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.671135    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qg9x\" (UniqueName: \"kubernetes.io/projected/0568c967-b6cc-458a-ab00-c056ef82630a-kube-api-access-7qg9x\") pod \"data-science-collector-collector-0\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.677065 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.674591    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.677538 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.677490    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.678369 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.678347    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0568c967-b6cc-458a-ab00-c056ef82630a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.679451 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.679427    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.680173 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.680153    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.693122 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.693094    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qg9x\" (UniqueName: \"kubernetes.io/projected/0568c967-b6cc-458a-ab00-c056ef82630a-kube-api-access-7qg9x\") pod \"data-science-collector-collector-0\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:26.744908 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:26.744888    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:27.034754 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:27.034730    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:27.036995 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:16:27.036969    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0568c967_b6cc_458a_ab00_c056ef82630a.slice/crio-5ad9474a20d19fe9089b1066262f18f9c706afa0ce43e4c51389aa33b0ba030b WatchSource:0}: Error finding container 5ad9474a20d19fe9089b1066262f18f9c706afa0ce43e4c51389aa33b0ba030b: Status 404 returned error can't find the container with id 5ad9474a20d19fe9089b1066262f18f9c706afa0ce43e4c51389aa33b0ba030b
Oct 30 03:16:27.039206 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:27.039141    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:16:27.039299 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:27.039225    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:16:27.342409 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:27.342374    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0568c967-b6cc-458a-ab00-c056ef82630a","Type":"ContainerStarted","Data":"df58880b86dd96f8d605d671f2bad13f935a79b516160970b13862ea10b3d7bd"}
Oct 30 03:16:27.342595 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:27.342429    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0568c967-b6cc-458a-ab00-c056ef82630a","Type":"ContainerStarted","Data":"5ad9474a20d19fe9089b1066262f18f9c706afa0ce43e4c51389aa33b0ba030b"}
Oct 30 03:16:27.386055 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:27.386006    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.385992868 podStartE2EDuration="1.385992868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:27.384958322 +0000 UTC m=+2068.034006763" watchObservedRunningTime="2026-10-30 03:16:27.385992868 +0000 UTC m=+2068.035041359"
Oct 30 03:16:27.502907 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:27.502878    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a17e5b2-a757-498e-b825-2bd9ee4f7fbe" path="/var/lib/kubelet/pods/3a17e5b2-a757-498e-b825-2bd9ee4f7fbe/volumes"
Oct 30 03:16:28.913791 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:28.913752    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:29.118253 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:29.118223    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:29.227416 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:29.227343    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:30.927626 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:30.927589    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:31.047644 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:31.047609    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:32.731956 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:32.731921    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:32.732488 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:32.732100    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="0568c967-b6cc-458a-ab00-c056ef82630a" containerName="otc-container" containerID="cri-o://df58880b86dd96f8d605d671f2bad13f935a79b516160970b13862ea10b3d7bd" gracePeriod=30
Oct 30 03:16:33.983517 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:33.983494    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.029384 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.029354    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-monitoring-tls-certs\") pod \"0568c967-b6cc-458a-ab00-c056ef82630a\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") "
Oct 30 03:16:34.029530 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.029412    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-otlp-tls-certs\") pod \"0568c967-b6cc-458a-ab00-c056ef82630a\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") "
Oct 30 03:16:34.029530 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.029436    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0568c967-b6cc-458a-ab00-c056ef82630a-otc-internal\" (UniqueName: \"kubernetes.io/configmap/0568c967-b6cc-458a-ab00-c056ef82630a-otc-internal\") pod \"0568c967-b6cc-458a-ab00-c056ef82630a\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") "
Oct 30 03:16:34.029530 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.029478    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-tls-certs\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-tls-certs\") pod \"0568c967-b6cc-458a-ab00-c056ef82630a\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") "
Oct 30 03:16:34.029701 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.029578    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-data-science-collector-ta-client-cert\") pod \"0568c967-b6cc-458a-ab00-c056ef82630a\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") "
Oct 30 03:16:34.029875 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.029841    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0568c967-b6cc-458a-ab00-c056ef82630a-otc-internal" pod "0568c967-b6cc-458a-ab00-c056ef82630a" (UID: "0568c967-b6cc-458a-ab00-c056ef82630a"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:34.031792 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.031770    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-otlp-tls-certs" pod "0568c967-b6cc-458a-ab00-c056ef82630a" (UID: "0568c967-b6cc-458a-ab00-c056ef82630a"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:34.031923 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.031796    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-monitoring-tls-certs" pod "0568c967-b6cc-458a-ab00-c056ef82630a" (UID: "0568c967-b6cc-458a-ab00-c056ef82630a"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:34.031923 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.031889    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-data-science-collector-ta-client-cert" pod "0568c967-b6cc-458a-ab00-c056ef82630a" (UID: "0568c967-b6cc-458a-ab00-c056ef82630a"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:34.032040 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.031962    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-tls-certs" pod "0568c967-b6cc-458a-ab00-c056ef82630a" (UID: "0568c967-b6cc-458a-ab00-c056ef82630a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:34.130739 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.130705    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0568c967-b6cc-458a-ab00-c056ef82630a-kube-api-access-7qg9x\" (UniqueName: \"kubernetes.io/projected/0568c967-b6cc-458a-ab00-c056ef82630a-kube-api-access-7qg9x\") pod \"0568c967-b6cc-458a-ab00-c056ef82630a\" (UID: \"0568c967-b6cc-458a-ab00-c056ef82630a\") "
Oct 30 03:16:34.130928 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.130915    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:34.130997 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.130930    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:34.130997 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.130940    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0568c967-b6cc-458a-ab00-c056ef82630a-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:34.130997 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.130948    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:34.130997 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.130957    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0568c967-b6cc-458a-ab00-c056ef82630a-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:34.132705 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.132682    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0568c967-b6cc-458a-ab00-c056ef82630a-kube-api-access-7qg9x" pod "0568c967-b6cc-458a-ab00-c056ef82630a" (UID: "0568c967-b6cc-458a-ab00-c056ef82630a"). InnerVolumeSpecName "kube-api-access-7qg9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:34.232058 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.232031    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qg9x\" (UniqueName: \"kubernetes.io/projected/0568c967-b6cc-458a-ab00-c056ef82630a-kube-api-access-7qg9x\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:34.376715 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.376680    4960 generic.go:356] "Generic (PLEG): container finished" podID="0568c967-b6cc-458a-ab00-c056ef82630a" containerID="df58880b86dd96f8d605d671f2bad13f935a79b516160970b13862ea10b3d7bd" exitCode=0
Oct 30 03:16:34.376835 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.376734    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.376835 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.376767    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0568c967-b6cc-458a-ab00-c056ef82630a","Type":"ContainerDied","Data":"df58880b86dd96f8d605d671f2bad13f935a79b516160970b13862ea10b3d7bd"}
Oct 30 03:16:34.376835 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.376819    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0568c967-b6cc-458a-ab00-c056ef82630a","Type":"ContainerDied","Data":"5ad9474a20d19fe9089b1066262f18f9c706afa0ce43e4c51389aa33b0ba030b"}
Oct 30 03:16:34.376996 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.376842    4960 scope.go:122] "RemoveContainer" containerID="df58880b86dd96f8d605d671f2bad13f935a79b516160970b13862ea10b3d7bd"
Oct 30 03:16:34.386095 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.386081    4960 scope.go:122] "RemoveContainer" containerID="df58880b86dd96f8d605d671f2bad13f935a79b516160970b13862ea10b3d7bd"
Oct 30 03:16:34.386393 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:16:34.386377    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df58880b86dd96f8d605d671f2bad13f935a79b516160970b13862ea10b3d7bd\": container with ID starting with df58880b86dd96f8d605d671f2bad13f935a79b516160970b13862ea10b3d7bd not found: ID does not exist" containerID="df58880b86dd96f8d605d671f2bad13f935a79b516160970b13862ea10b3d7bd"
Oct 30 03:16:34.386470 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.386399    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df58880b86dd96f8d605d671f2bad13f935a79b516160970b13862ea10b3d7bd"} err="failed to get container status \"df58880b86dd96f8d605d671f2bad13f935a79b516160970b13862ea10b3d7bd\": rpc error: code = NotFound desc = could not find container \"df58880b86dd96f8d605d671f2bad13f935a79b516160970b13862ea10b3d7bd\": container with ID starting with df58880b86dd96f8d605d671f2bad13f935a79b516160970b13862ea10b3d7bd not found: ID does not exist"
Oct 30 03:16:34.430051 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.430030    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:34.443608 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.443585    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:34.470267 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.470240    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:34.470623 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.470610    4960 cpu_manager.go:414] "containerMap: removing container" podUID="0568c967-b6cc-458a-ab00-c056ef82630a" containerName="otc-container"
Oct 30 03:16:34.470665 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.470625    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0568c967-b6cc-458a-ab00-c056ef82630a" containerName="otc-container"
Oct 30 03:16:34.470736 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.470718    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="0568c967-b6cc-458a-ab00-c056ef82630a" containerName="otc-container"
Oct 30 03:16:34.475334 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.475317    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.493092 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.493058    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:16:34.493092 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.493067    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:16:34.493267 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.493097    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:16:34.493267 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.493162    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:16:34.493267 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.493171    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:16:34.493423 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.493283    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:16:34.494623 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.494601    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:34.534027 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.534005    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.534128 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.534037    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d21df132-4d83-497f-8480-d051b132ebb0-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.534128 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.534055    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.534128 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.534077    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.534310 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.534291    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.534371 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.534356    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75hbf\" (UniqueName: \"kubernetes.io/projected/d21df132-4d83-497f-8480-d051b132ebb0-kube-api-access-75hbf\") pod \"data-science-collector-collector-0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.635030 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.634975    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.635030 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.635005    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d21df132-4d83-497f-8480-d051b132ebb0-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.635030 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.635024    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.635293 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.635040    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.635293 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.635067    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.635293 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.635112    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75hbf\" (UniqueName: \"kubernetes.io/projected/d21df132-4d83-497f-8480-d051b132ebb0-kube-api-access-75hbf\") pod \"data-science-collector-collector-0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.635753 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.635726    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d21df132-4d83-497f-8480-d051b132ebb0-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.637691 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.637671    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.637779 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.637705    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.638153 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.638133    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.638289 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.638141    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.655399 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.655374    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75hbf\" (UniqueName: \"kubernetes.io/projected/d21df132-4d83-497f-8480-d051b132ebb0-kube-api-access-75hbf\") pod \"data-science-collector-collector-0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:34.787150 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:34.787127    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:35.083641 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:35.083617    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:35.085223 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:16:35.085197    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21df132_4d83_497f_8480_d051b132ebb0.slice/crio-9f78c825cad3c72aca3d917184939c899baa8cd0ee116b8d1b42b85ec5224b9d WatchSource:0}: Error finding container 9f78c825cad3c72aca3d917184939c899baa8cd0ee116b8d1b42b85ec5224b9d: Status 404 returned error can't find the container with id 9f78c825cad3c72aca3d917184939c899baa8cd0ee116b8d1b42b85ec5224b9d
Oct 30 03:16:35.087966 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:35.087927    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:16:35.088057 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:35.088011    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:16:35.384396 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:35.384310    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d21df132-4d83-497f-8480-d051b132ebb0","Type":"ContainerStarted","Data":"88da6d1e3484bb0e62442499f75fb56627f13d4ebebea8e4812dc5a0c1a2e85b"}
Oct 30 03:16:35.384396 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:35.384373    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d21df132-4d83-497f-8480-d051b132ebb0","Type":"ContainerStarted","Data":"9f78c825cad3c72aca3d917184939c899baa8cd0ee116b8d1b42b85ec5224b9d"}
Oct 30 03:16:35.426877 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:35.426833    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.426819943 podStartE2EDuration="1.426819943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:35.42572129 +0000 UTC m=+2076.074769715" watchObservedRunningTime="2026-10-30 03:16:35.426819943 +0000 UTC m=+2076.075868364"
Oct 30 03:16:35.503614 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:35.503567    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0568c967-b6cc-458a-ab00-c056ef82630a" path="/var/lib/kubelet/pods/0568c967-b6cc-458a-ab00-c056ef82630a/volumes"
Oct 30 03:16:37.082459 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:37.082425    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:37.201296 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:37.201262    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:37.325688 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:37.325662    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:37.436342 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:37.436267    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:37.784940 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:37.784862    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:38.995240 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:38.995193    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:39.117561 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:39.117528    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:47.841019 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:47.840982    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:47.841557 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:47.841201    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="d21df132-4d83-497f-8480-d051b132ebb0" containerName="otc-container" containerID="cri-o://88da6d1e3484bb0e62442499f75fb56627f13d4ebebea8e4812dc5a0c1a2e85b" gracePeriod=30
Oct 30 03:16:49.086801 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.086775    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.151459 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.151425    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-tls-certs\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-tls-certs\") pod \"d21df132-4d83-497f-8480-d051b132ebb0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") "
Oct 30 03:16:49.151641 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.151480    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-otlp-tls-certs\") pod \"d21df132-4d83-497f-8480-d051b132ebb0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") "
Oct 30 03:16:49.151641 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.151517    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d21df132-4d83-497f-8480-d051b132ebb0-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d21df132-4d83-497f-8480-d051b132ebb0-otc-internal\") pod \"d21df132-4d83-497f-8480-d051b132ebb0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") "
Oct 30 03:16:49.151641 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.151615    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-data-science-collector-ta-client-cert\") pod \"d21df132-4d83-497f-8480-d051b132ebb0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") "
Oct 30 03:16:49.151810 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.151659    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-monitoring-tls-certs\") pod \"d21df132-4d83-497f-8480-d051b132ebb0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") "
Oct 30 03:16:49.151810 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.151693    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d21df132-4d83-497f-8480-d051b132ebb0-kube-api-access-75hbf\" (UniqueName: \"kubernetes.io/projected/d21df132-4d83-497f-8480-d051b132ebb0-kube-api-access-75hbf\") pod \"d21df132-4d83-497f-8480-d051b132ebb0\" (UID: \"d21df132-4d83-497f-8480-d051b132ebb0\") "
Oct 30 03:16:49.151960 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.151933    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d21df132-4d83-497f-8480-d051b132ebb0-otc-internal" pod "d21df132-4d83-497f-8480-d051b132ebb0" (UID: "d21df132-4d83-497f-8480-d051b132ebb0"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:49.154204 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.154142    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-otlp-tls-certs" pod "d21df132-4d83-497f-8480-d051b132ebb0" (UID: "d21df132-4d83-497f-8480-d051b132ebb0"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:49.154505 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.154469    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-tls-certs" pod "d21df132-4d83-497f-8480-d051b132ebb0" (UID: "d21df132-4d83-497f-8480-d051b132ebb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:49.154616 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.154516    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-data-science-collector-ta-client-cert" pod "d21df132-4d83-497f-8480-d051b132ebb0" (UID: "d21df132-4d83-497f-8480-d051b132ebb0"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:49.154730 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.154703    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-monitoring-tls-certs" pod "d21df132-4d83-497f-8480-d051b132ebb0" (UID: "d21df132-4d83-497f-8480-d051b132ebb0"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:49.154862 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.154845    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21df132-4d83-497f-8480-d051b132ebb0-kube-api-access-75hbf" pod "d21df132-4d83-497f-8480-d051b132ebb0" (UID: "d21df132-4d83-497f-8480-d051b132ebb0"). InnerVolumeSpecName "kube-api-access-75hbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:49.252829 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.252774    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:49.252829 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.252804    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:49.252829 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.252815    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75hbf\" (UniqueName: \"kubernetes.io/projected/d21df132-4d83-497f-8480-d051b132ebb0-kube-api-access-75hbf\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:49.252829 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.252824    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:49.252829 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.252832    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d21df132-4d83-497f-8480-d051b132ebb0-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:49.253040 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.252840    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d21df132-4d83-497f-8480-d051b132ebb0-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:16:49.452407 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.452380    4960 generic.go:356] "Generic (PLEG): container finished" podID="d21df132-4d83-497f-8480-d051b132ebb0" containerID="88da6d1e3484bb0e62442499f75fb56627f13d4ebebea8e4812dc5a0c1a2e85b" exitCode=0
Oct 30 03:16:49.452547 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.452424    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.452547 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.452464    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d21df132-4d83-497f-8480-d051b132ebb0","Type":"ContainerDied","Data":"88da6d1e3484bb0e62442499f75fb56627f13d4ebebea8e4812dc5a0c1a2e85b"}
Oct 30 03:16:49.452547 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.452522    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d21df132-4d83-497f-8480-d051b132ebb0","Type":"ContainerDied","Data":"9f78c825cad3c72aca3d917184939c899baa8cd0ee116b8d1b42b85ec5224b9d"}
Oct 30 03:16:49.452547 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.452546    4960 scope.go:122] "RemoveContainer" containerID="88da6d1e3484bb0e62442499f75fb56627f13d4ebebea8e4812dc5a0c1a2e85b"
Oct 30 03:16:49.462395 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.462369    4960 scope.go:122] "RemoveContainer" containerID="88da6d1e3484bb0e62442499f75fb56627f13d4ebebea8e4812dc5a0c1a2e85b"
Oct 30 03:16:49.462687 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:16:49.462668    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88da6d1e3484bb0e62442499f75fb56627f13d4ebebea8e4812dc5a0c1a2e85b\": container with ID starting with 88da6d1e3484bb0e62442499f75fb56627f13d4ebebea8e4812dc5a0c1a2e85b not found: ID does not exist" containerID="88da6d1e3484bb0e62442499f75fb56627f13d4ebebea8e4812dc5a0c1a2e85b"
Oct 30 03:16:49.462775 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.462697    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88da6d1e3484bb0e62442499f75fb56627f13d4ebebea8e4812dc5a0c1a2e85b"} err="failed to get container status \"88da6d1e3484bb0e62442499f75fb56627f13d4ebebea8e4812dc5a0c1a2e85b\": rpc error: code = NotFound desc = could not find container \"88da6d1e3484bb0e62442499f75fb56627f13d4ebebea8e4812dc5a0c1a2e85b\": container with ID starting with 88da6d1e3484bb0e62442499f75fb56627f13d4ebebea8e4812dc5a0c1a2e85b not found: ID does not exist"
Oct 30 03:16:49.510237 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.510170    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:49.523564 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.523538    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:49.549098 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.549073    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:49.549468 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.549454    4960 cpu_manager.go:414] "containerMap: removing container" podUID="d21df132-4d83-497f-8480-d051b132ebb0" containerName="otc-container"
Oct 30 03:16:49.549522 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.549470    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d21df132-4d83-497f-8480-d051b132ebb0" containerName="otc-container"
Oct 30 03:16:49.549557 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.549536    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="d21df132-4d83-497f-8480-d051b132ebb0" containerName="otc-container"
Oct 30 03:16:49.552398 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.552381    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.571926 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.571795    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:16:49.571926 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.571827    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:16:49.571926 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.571856    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:16:49.572092 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.571963    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:16:49.572092 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.572048    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:16:49.572212 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.572144    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:16:49.573153 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.573133    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:49.656943 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.656919    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.657047 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.656964    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.657047 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.656988    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f46a9b92-ed33-458e-b267-471a25451152-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.657136 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.657078    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqnfr\" (UniqueName: \"kubernetes.io/projected/f46a9b92-ed33-458e-b267-471a25451152-kube-api-access-lqnfr\") pod \"data-science-collector-collector-0\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.657136 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.657105    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.657136 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.657132    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.758117 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.758095    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.758226 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.758126    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.758226 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.758160    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.758226 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.758212    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.758335 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.758233    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f46a9b92-ed33-458e-b267-471a25451152-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.758335 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.758254    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqnfr\" (UniqueName: \"kubernetes.io/projected/f46a9b92-ed33-458e-b267-471a25451152-kube-api-access-lqnfr\") pod \"data-science-collector-collector-0\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.759159 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.759130    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f46a9b92-ed33-458e-b267-471a25451152-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.760816 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.760763    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.760891 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.760870    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.760949 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.760929    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.761025 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.761011    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.779961 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.779943    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqnfr\" (UniqueName: \"kubernetes.io/projected/f46a9b92-ed33-458e-b267-471a25451152-kube-api-access-lqnfr\") pod \"data-science-collector-collector-0\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:49.870939 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:49.870921    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:16:50.162307 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:50.162282    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:16:50.163346 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:16:50.163324    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46a9b92_ed33_458e_b267_471a25451152.slice/crio-9746026b6475facacfe1b905ad58e6316fee1324ed87bce6ca141c488e61692f WatchSource:0}: Error finding container 9746026b6475facacfe1b905ad58e6316fee1324ed87bce6ca141c488e61692f: Status 404 returned error can't find the container with id 9746026b6475facacfe1b905ad58e6316fee1324ed87bce6ca141c488e61692f
Oct 30 03:16:50.165210 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:50.165161    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:16:50.165291 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:50.165243    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:16:50.459082 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:50.458983    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f46a9b92-ed33-458e-b267-471a25451152","Type":"ContainerStarted","Data":"721395c3deb77558c8a81b53d6b60a38e5873a0488efd8949b0c78d1c2a47775"}
Oct 30 03:16:50.459082 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:50.459067    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f46a9b92-ed33-458e-b267-471a25451152","Type":"ContainerStarted","Data":"9746026b6475facacfe1b905ad58e6316fee1324ed87bce6ca141c488e61692f"}
Oct 30 03:16:50.505313 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:50.505254    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.505240039 podStartE2EDuration="1.505240039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:50.503789078 +0000 UTC m=+2091.152837502" watchObservedRunningTime="2026-10-30 03:16:50.505240039 +0000 UTC m=+2091.154288459"
Oct 30 03:16:51.503335 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:51.503301    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d21df132-4d83-497f-8480-d051b132ebb0" path="/var/lib/kubelet/pods/d21df132-4d83-497f-8480-d051b132ebb0/volumes"
Oct 30 03:16:53.417092 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:53.417063    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:53.953635 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:16:53.953603    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:16:59.680799 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:59.680770    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:16:59.691639 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:59.686305    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:16:59.693372 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:59.693343    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:16:59.698592 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:16:59.698573    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:17:23.953215 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:17:23.953129    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:17:53.543160 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:17:53.543126    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:53.953206 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:17:53.953160    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:18:23.953811 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:18:23.953772    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:18:53.656989 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:18:53.656952    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:53.764599 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:18:53.764571    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:53.953633 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:18:53.953556    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:19:11.518235 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:19:11.518204    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:23.952631 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:19:23.952591    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:19:53.775402 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:19:53.775368    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:53.954733 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:19:53.954692    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:20:23.954236 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:20:23.954140    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:20:27.846884 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:20:27.846846    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:53.877915 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:20:53.877883    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:53.952557 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:20:53.952506    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:21:09.355653 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:21:09.355613    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:23.953334 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:21:23.953300    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:21:53.954103 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:21:53.954067    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:21:54.032440 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:21:54.032402    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:59.716272 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:21:59.716244    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:21:59.721456 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:21:59.721430    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:21:59.722520 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:21:59.722500    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:21:59.727388 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:21:59.727374    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:22:23.953692 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:22:23.953651    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:22:32.457807 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:22:32.457774    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:32.575438 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:22:32.575409    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:53.957061 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:22:53.957010    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:22:54.159732 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:22:54.159696    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:54.298853 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:22:54.298788    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:23.957103 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:23:23.957009    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:23:42.834256 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:23:42.834217    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:53.953643 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:23:53.953611    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:23:54.396798 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:23:54.396771    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:23.955148 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:24:23.955105    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:24:53.092666 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:24:53.092630    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:53.953482 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:24:53.953450    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:24:54.410555 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:24:54.410523    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:23.953621 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:25:23.953581    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:25:53.954421 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:25:53.954384    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:25:54.583891 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:25:54.583848    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:23.956335 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:26:23.956256    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:26:44.430422 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:44.430388    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:44.552735 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:44.552668    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:44.733376 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:44.733300    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:44.936362 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:44.936326    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:46.688964 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:46.688933    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:46.829457 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:46.829421    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:49.566875 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:49.566841    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:49.700342 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:49.700303    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:50.241887 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:50.241843    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:50.668646 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:50.668616    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:26:50.669096 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:50.668843    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="f46a9b92-ed33-458e-b267-471a25451152" containerName="otc-container" containerID="cri-o://721395c3deb77558c8a81b53d6b60a38e5873a0488efd8949b0c78d1c2a47775" gracePeriod=30
Oct 30 03:26:53.019390 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.019366    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.111547 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.111519    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-data-science-collector-ta-client-cert\") pod \"f46a9b92-ed33-458e-b267-471a25451152\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") "
Oct 30 03:26:53.111674 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.111565    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-monitoring-tls-certs\") pod \"f46a9b92-ed33-458e-b267-471a25451152\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") "
Oct 30 03:26:53.111674 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.111599    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-tls-certs\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-tls-certs\") pod \"f46a9b92-ed33-458e-b267-471a25451152\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") "
Oct 30 03:26:53.111674 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.111637    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46a9b92-ed33-458e-b267-471a25451152-kube-api-access-lqnfr\" (UniqueName: \"kubernetes.io/projected/f46a9b92-ed33-458e-b267-471a25451152-kube-api-access-lqnfr\") pod \"f46a9b92-ed33-458e-b267-471a25451152\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") "
Oct 30 03:26:53.111844 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.111673    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46a9b92-ed33-458e-b267-471a25451152-otc-internal\" (UniqueName: \"kubernetes.io/configmap/f46a9b92-ed33-458e-b267-471a25451152-otc-internal\") pod \"f46a9b92-ed33-458e-b267-471a25451152\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") "
Oct 30 03:26:53.111844 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.111711    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-otlp-tls-certs\") pod \"f46a9b92-ed33-458e-b267-471a25451152\" (UID: \"f46a9b92-ed33-458e-b267-471a25451152\") "
Oct 30 03:26:53.112080 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.112053    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46a9b92-ed33-458e-b267-471a25451152-otc-internal" pod "f46a9b92-ed33-458e-b267-471a25451152" (UID: "f46a9b92-ed33-458e-b267-471a25451152"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:53.113901 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.113872    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-otlp-tls-certs" pod "f46a9b92-ed33-458e-b267-471a25451152" (UID: "f46a9b92-ed33-458e-b267-471a25451152"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:53.114011 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.113947    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46a9b92-ed33-458e-b267-471a25451152-kube-api-access-lqnfr" pod "f46a9b92-ed33-458e-b267-471a25451152" (UID: "f46a9b92-ed33-458e-b267-471a25451152"). InnerVolumeSpecName "kube-api-access-lqnfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:53.114011 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.113984    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-tls-certs" pod "f46a9b92-ed33-458e-b267-471a25451152" (UID: "f46a9b92-ed33-458e-b267-471a25451152"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:53.114270 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.114252    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-monitoring-tls-certs" pod "f46a9b92-ed33-458e-b267-471a25451152" (UID: "f46a9b92-ed33-458e-b267-471a25451152"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:53.114327 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.114293    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-data-science-collector-ta-client-cert" pod "f46a9b92-ed33-458e-b267-471a25451152" (UID: "f46a9b92-ed33-458e-b267-471a25451152"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:53.213057 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.213031    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqnfr\" (UniqueName: \"kubernetes.io/projected/f46a9b92-ed33-458e-b267-471a25451152-kube-api-access-lqnfr\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:26:53.213147 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.213062    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f46a9b92-ed33-458e-b267-471a25451152-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:26:53.213147 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.213077    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:26:53.213147 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.213092    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:26:53.213147 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.213108    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:26:53.213147 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.213123    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f46a9b92-ed33-458e-b267-471a25451152-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:26:53.216866 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.216841    4960 generic.go:356] "Generic (PLEG): container finished" podID="f46a9b92-ed33-458e-b267-471a25451152" containerID="721395c3deb77558c8a81b53d6b60a38e5873a0488efd8949b0c78d1c2a47775" exitCode=0
Oct 30 03:26:53.216951 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.216885    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.216951 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.216888    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f46a9b92-ed33-458e-b267-471a25451152","Type":"ContainerDied","Data":"721395c3deb77558c8a81b53d6b60a38e5873a0488efd8949b0c78d1c2a47775"}
Oct 30 03:26:53.216951 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.216929    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f46a9b92-ed33-458e-b267-471a25451152","Type":"ContainerDied","Data":"9746026b6475facacfe1b905ad58e6316fee1324ed87bce6ca141c488e61692f"}
Oct 30 03:26:53.216951 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.216950    4960 scope.go:122] "RemoveContainer" containerID="721395c3deb77558c8a81b53d6b60a38e5873a0488efd8949b0c78d1c2a47775"
Oct 30 03:26:53.225936 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.225918    4960 scope.go:122] "RemoveContainer" containerID="721395c3deb77558c8a81b53d6b60a38e5873a0488efd8949b0c78d1c2a47775"
Oct 30 03:26:53.226241 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:26:53.226224    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"721395c3deb77558c8a81b53d6b60a38e5873a0488efd8949b0c78d1c2a47775\": container with ID starting with 721395c3deb77558c8a81b53d6b60a38e5873a0488efd8949b0c78d1c2a47775 not found: ID does not exist" containerID="721395c3deb77558c8a81b53d6b60a38e5873a0488efd8949b0c78d1c2a47775"
Oct 30 03:26:53.226313 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.226248    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"721395c3deb77558c8a81b53d6b60a38e5873a0488efd8949b0c78d1c2a47775"} err="failed to get container status \"721395c3deb77558c8a81b53d6b60a38e5873a0488efd8949b0c78d1c2a47775\": rpc error: code = NotFound desc = could not find container \"721395c3deb77558c8a81b53d6b60a38e5873a0488efd8949b0c78d1c2a47775\": container with ID starting with 721395c3deb77558c8a81b53d6b60a38e5873a0488efd8949b0c78d1c2a47775 not found: ID does not exist"
Oct 30 03:26:53.274393 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.274373    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:26:53.283523 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.279874    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:26:53.328565 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.328541    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:26:53.328909 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.328897    4960 cpu_manager.go:414] "containerMap: removing container" podUID="f46a9b92-ed33-458e-b267-471a25451152" containerName="otc-container"
Oct 30 03:26:53.328954 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.328911    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46a9b92-ed33-458e-b267-471a25451152" containerName="otc-container"
Oct 30 03:26:53.328988 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.328966    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46a9b92-ed33-458e-b267-471a25451152" containerName="otc-container"
Oct 30 03:26:53.333386 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.333372    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.358091 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.358064    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:26:53.358742 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.358711    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:26:53.358846 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.358794    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:26:53.358918 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.358897    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:26:53.358972 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.358934    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:26:53.359024 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.358999    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:26:53.359953 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.359937    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:26:53.414135 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.414090    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.414135 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.414128    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.414280 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.414215    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcgnd\" (UniqueName: \"kubernetes.io/projected/7631744a-c7d3-4202-9a2e-20c03e4e0521-kube-api-access-rcgnd\") pod \"data-science-collector-collector-0\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.414280 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.414250    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.414280 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.414272    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.414422 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.414400    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7631744a-c7d3-4202-9a2e-20c03e4e0521-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.502190 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.502153    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f46a9b92-ed33-458e-b267-471a25451152" path="/var/lib/kubelet/pods/f46a9b92-ed33-458e-b267-471a25451152/volumes"
Oct 30 03:26:53.515668 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.515646    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcgnd\" (UniqueName: \"kubernetes.io/projected/7631744a-c7d3-4202-9a2e-20c03e4e0521-kube-api-access-rcgnd\") pod \"data-science-collector-collector-0\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.515764 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.515677    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.515764 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.515698    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.515764 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.515734    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7631744a-c7d3-4202-9a2e-20c03e4e0521-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.515924 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.515789    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.515924 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.515828    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.516500 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.516478    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7631744a-c7d3-4202-9a2e-20c03e4e0521-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.518280 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.518247    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.518411 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.518393    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.518720 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.518696    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.518761 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.518740    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.537745 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.537724    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcgnd\" (UniqueName: \"kubernetes.io/projected/7631744a-c7d3-4202-9a2e-20c03e4e0521-kube-api-access-rcgnd\") pod \"data-science-collector-collector-0\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.646136 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.646115    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:26:53.940469 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.940443    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:26:53.942146 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:26:53.942115    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7631744a_c7d3_4202_9a2e_20c03e4e0521.slice/crio-3b1a6c65918ee2c49289d0f3bd4bb833c6b6dc5c1666c55896db114fa6a98e25 WatchSource:0}: Error finding container 3b1a6c65918ee2c49289d0f3bd4bb833c6b6dc5c1666c55896db114fa6a98e25: Status 404 returned error can't find the container with id 3b1a6c65918ee2c49289d0f3bd4bb833c6b6dc5c1666c55896db114fa6a98e25
Oct 30 03:26:53.944425 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.944391    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:26:53.944529 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:53.944468    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:26:53.955127 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:26:53.955055    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:26:54.224051 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:54.223958    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7631744a-c7d3-4202-9a2e-20c03e4e0521","Type":"ContainerStarted","Data":"9820383fc9070c4c843a010c9988f604b6b1490661141172e292adaf61b5684a"}
Oct 30 03:26:54.224051 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:54.224023    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7631744a-c7d3-4202-9a2e-20c03e4e0521","Type":"ContainerStarted","Data":"3b1a6c65918ee2c49289d0f3bd4bb833c6b6dc5c1666c55896db114fa6a98e25"}
Oct 30 03:26:54.271831 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:54.271789    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.271776364 podStartE2EDuration="1.271776364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:54.27008159 +0000 UTC m=+2694.919130016" watchObservedRunningTime="2026-10-30 03:26:54.271776364 +0000 UTC m=+2694.920824782"
Oct 30 03:26:54.648125 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:54.648090    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:56.524923 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:56.524885    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:56.744731 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:56.744701    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:56.986850 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:56.986816    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:57.438668 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:57.438634    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:57.608820 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:57.608795    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:58.688725 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:58.688687    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:58.822605 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:58.822520    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:59.745688 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:59.745663    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:26:59.752377 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:59.752352    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:26:59.752377 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:59.752363    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:26:59.758083 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:26:59.758065    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:27:00.198324 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:00.198293    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:00.329536 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:00.329505    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:01.734250 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:01.734201    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:27:01.734737 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:01.734473    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="7631744a-c7d3-4202-9a2e-20c03e4e0521" containerName="otc-container" containerID="cri-o://9820383fc9070c4c843a010c9988f604b6b1490661141172e292adaf61b5684a" gracePeriod=30
Oct 30 03:27:01.987685 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:01.987629    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.085409 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.085382    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-tls-certs\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-tls-certs\") pod \"7631744a-c7d3-4202-9a2e-20c03e4e0521\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") "
Oct 30 03:27:02.085547 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.085437    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-data-science-collector-ta-client-cert\") pod \"7631744a-c7d3-4202-9a2e-20c03e4e0521\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") "
Oct 30 03:27:02.085547 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.085495    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7631744a-c7d3-4202-9a2e-20c03e4e0521-kube-api-access-rcgnd\" (UniqueName: \"kubernetes.io/projected/7631744a-c7d3-4202-9a2e-20c03e4e0521-kube-api-access-rcgnd\") pod \"7631744a-c7d3-4202-9a2e-20c03e4e0521\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") "
Oct 30 03:27:02.085635 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.085557    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-monitoring-tls-certs\") pod \"7631744a-c7d3-4202-9a2e-20c03e4e0521\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") "
Oct 30 03:27:02.085635 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.085582    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-otlp-tls-certs\") pod \"7631744a-c7d3-4202-9a2e-20c03e4e0521\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") "
Oct 30 03:27:02.085635 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.085606    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7631744a-c7d3-4202-9a2e-20c03e4e0521-otc-internal\" (UniqueName: \"kubernetes.io/configmap/7631744a-c7d3-4202-9a2e-20c03e4e0521-otc-internal\") pod \"7631744a-c7d3-4202-9a2e-20c03e4e0521\" (UID: \"7631744a-c7d3-4202-9a2e-20c03e4e0521\") "
Oct 30 03:27:02.086312 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.086277    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7631744a-c7d3-4202-9a2e-20c03e4e0521-otc-internal" pod "7631744a-c7d3-4202-9a2e-20c03e4e0521" (UID: "7631744a-c7d3-4202-9a2e-20c03e4e0521"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:02.087785 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.087757    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7631744a-c7d3-4202-9a2e-20c03e4e0521-kube-api-access-rcgnd" pod "7631744a-c7d3-4202-9a2e-20c03e4e0521" (UID: "7631744a-c7d3-4202-9a2e-20c03e4e0521"). InnerVolumeSpecName "kube-api-access-rcgnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:02.087895 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.087804    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-monitoring-tls-certs" pod "7631744a-c7d3-4202-9a2e-20c03e4e0521" (UID: "7631744a-c7d3-4202-9a2e-20c03e4e0521"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:02.087895 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.087852    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-tls-certs" pod "7631744a-c7d3-4202-9a2e-20c03e4e0521" (UID: "7631744a-c7d3-4202-9a2e-20c03e4e0521"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:02.088288 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.088267    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-otlp-tls-certs" pod "7631744a-c7d3-4202-9a2e-20c03e4e0521" (UID: "7631744a-c7d3-4202-9a2e-20c03e4e0521"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:02.088336 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.088281    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-data-science-collector-ta-client-cert" pod "7631744a-c7d3-4202-9a2e-20c03e4e0521" (UID: "7631744a-c7d3-4202-9a2e-20c03e4e0521"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:02.186811 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.186789    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:27:02.186811 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.186809    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:27:02.186941 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.186819    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7631744a-c7d3-4202-9a2e-20c03e4e0521-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:27:02.186941 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.186827    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:27:02.186941 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.186836    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7631744a-c7d3-4202-9a2e-20c03e4e0521-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:27:02.186941 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.186845    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcgnd\" (UniqueName: \"kubernetes.io/projected/7631744a-c7d3-4202-9a2e-20c03e4e0521-kube-api-access-rcgnd\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:27:02.278521 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.278443    4960 generic.go:356] "Generic (PLEG): container finished" podID="7631744a-c7d3-4202-9a2e-20c03e4e0521" containerID="9820383fc9070c4c843a010c9988f604b6b1490661141172e292adaf61b5684a" exitCode=0
Oct 30 03:27:02.278521 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.278497    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.278710 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.278529    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7631744a-c7d3-4202-9a2e-20c03e4e0521","Type":"ContainerDied","Data":"9820383fc9070c4c843a010c9988f604b6b1490661141172e292adaf61b5684a"}
Oct 30 03:27:02.278710 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.278584    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7631744a-c7d3-4202-9a2e-20c03e4e0521","Type":"ContainerDied","Data":"3b1a6c65918ee2c49289d0f3bd4bb833c6b6dc5c1666c55896db114fa6a98e25"}
Oct 30 03:27:02.278710 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.278606    4960 scope.go:122] "RemoveContainer" containerID="9820383fc9070c4c843a010c9988f604b6b1490661141172e292adaf61b5684a"
Oct 30 03:27:02.288094 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.288078    4960 scope.go:122] "RemoveContainer" containerID="9820383fc9070c4c843a010c9988f604b6b1490661141172e292adaf61b5684a"
Oct 30 03:27:02.288380 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:27:02.288362    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9820383fc9070c4c843a010c9988f604b6b1490661141172e292adaf61b5684a\": container with ID starting with 9820383fc9070c4c843a010c9988f604b6b1490661141172e292adaf61b5684a not found: ID does not exist" containerID="9820383fc9070c4c843a010c9988f604b6b1490661141172e292adaf61b5684a"
Oct 30 03:27:02.288460 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.288387    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9820383fc9070c4c843a010c9988f604b6b1490661141172e292adaf61b5684a"} err="failed to get container status \"9820383fc9070c4c843a010c9988f604b6b1490661141172e292adaf61b5684a\": rpc error: code = NotFound desc = could not find container \"9820383fc9070c4c843a010c9988f604b6b1490661141172e292adaf61b5684a\": container with ID starting with 9820383fc9070c4c843a010c9988f604b6b1490661141172e292adaf61b5684a not found: ID does not exist"
Oct 30 03:27:02.343163 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.343138    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:27:02.359472 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.359449    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:27:02.409435 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.409414    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:27:02.409766 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.409754    4960 cpu_manager.go:414] "containerMap: removing container" podUID="7631744a-c7d3-4202-9a2e-20c03e4e0521" containerName="otc-container"
Oct 30 03:27:02.409812 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.409767    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7631744a-c7d3-4202-9a2e-20c03e4e0521" containerName="otc-container"
Oct 30 03:27:02.409850 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.409843    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="7631744a-c7d3-4202-9a2e-20c03e4e0521" containerName="otc-container"
Oct 30 03:27:02.414197 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.414164    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.440279 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.440257    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:27:02.440454 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.440439    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:27:02.440576 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.440560    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:27:02.449515 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.449494    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:27:02.449515 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.449509    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:27:02.449626 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.449514    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:27:02.471783 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.471763    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:27:02.489353 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.489328    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.489462 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.489363    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.489462 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.489443    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.489568 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.489499    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/21894725-c9c8-40c6-9c65-4a6724b9c80f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.489622 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.489576    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.489622 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.489604    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x5gk\" (UniqueName: \"kubernetes.io/projected/21894725-c9c8-40c6-9c65-4a6724b9c80f-kube-api-access-4x5gk\") pod \"data-science-collector-collector-0\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.590604 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.590582    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.590727 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.590610    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x5gk\" (UniqueName: \"kubernetes.io/projected/21894725-c9c8-40c6-9c65-4a6724b9c80f-kube-api-access-4x5gk\") pod \"data-science-collector-collector-0\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.590727 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.590634    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.590727 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.590657    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.590903 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.590782    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.590903 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.590852    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/21894725-c9c8-40c6-9c65-4a6724b9c80f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.591524 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.591505    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/21894725-c9c8-40c6-9c65-4a6724b9c80f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.593265 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.593246    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.593359 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.593278    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.593706 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.593690    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.593743 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.593696    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.615248 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.615228    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x5gk\" (UniqueName: \"kubernetes.io/projected/21894725-c9c8-40c6-9c65-4a6724b9c80f-kube-api-access-4x5gk\") pod \"data-science-collector-collector-0\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:02.726419 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:02.726399    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:27:03.019148 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:03.019119    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:27:03.020654 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:27:03.020611    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21894725_c9c8_40c6_9c65_4a6724b9c80f.slice/crio-cee5a53aa0b8ceadd555484711bc2058424ca4428a1916608a86c84d0e37855e WatchSource:0}: Error finding container cee5a53aa0b8ceadd555484711bc2058424ca4428a1916608a86c84d0e37855e: Status 404 returned error can't find the container with id cee5a53aa0b8ceadd555484711bc2058424ca4428a1916608a86c84d0e37855e
Oct 30 03:27:03.025710 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:03.025678    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:27:03.025816 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:03.025741    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:27:03.288014 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:03.287936    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"21894725-c9c8-40c6-9c65-4a6724b9c80f","Type":"ContainerStarted","Data":"6cac4c21ff7feec9e757cd8d4ba42f90d24219b1d0ac75bf6028cdce8add7214"}
Oct 30 03:27:03.288014 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:03.287984    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"21894725-c9c8-40c6-9c65-4a6724b9c80f","Type":"ContainerStarted","Data":"cee5a53aa0b8ceadd555484711bc2058424ca4428a1916608a86c84d0e37855e"}
Oct 30 03:27:03.333097 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:03.333044    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.333031355 podStartE2EDuration="1.333031355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:03.332155383 +0000 UTC m=+2703.981203806" watchObservedRunningTime="2026-10-30 03:27:03.333031355 +0000 UTC m=+2703.982079776"
Oct 30 03:27:03.503078 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:03.503039    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7631744a-c7d3-4202-9a2e-20c03e4e0521" path="/var/lib/kubelet/pods/7631744a-c7d3-4202-9a2e-20c03e4e0521/volumes"
Oct 30 03:27:23.953388 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:27:23.953350    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:27:53.953714 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:27:53.953676    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:27:54.744497 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:27:54.744465    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:23.952495 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:28:23.952461    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:28:37.342974 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:37.342938    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:37.464418 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:37.464386    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:43.264191 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:43.264154    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:43.393015 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:43.392977    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:43.525965 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:43.525891    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:49.095833 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:49.095798    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:49.313236 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:49.313201    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:49.723204 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:49.723125    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:49.746967 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:49.746933    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"]
Oct 30 03:28:49.764668 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:49.764646    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"
Oct 30 03:28:49.772291 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:49.772261    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"]
Oct 30 03:28:49.784445 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:49.784421    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-dcgm-https-18e330fa9f8b0f51\"/\"dcgm-https-exporter\""
Oct 30 03:28:49.784445 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:49.784430    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"tests-monitoring-dcgm-https-18e330fa9f8b0f51\"/\"default-dockercfg-nkkwx\""
Oct 30 03:28:49.784602 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:49.784431    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"tests-monitoring-dcgm-https-18e330fa9f8b0f51\"/\"dcgm-https-tls\""
Oct 30 03:28:49.855632 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:49.855603    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:49.903505 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:49.903483    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5b5d25c4-b1d0-493a-8c48-0105b692bd56-tls\") pod \"dcgm-https-exporter\" (UID: \"5b5d25c4-b1d0-493a-8c48-0105b692bd56\") " pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"
Oct 30 03:28:49.903626 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:49.903511    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5b5d25c4-b1d0-493a-8c48-0105b692bd56-data\") pod \"dcgm-https-exporter\" (UID: \"5b5d25c4-b1d0-493a-8c48-0105b692bd56\") " pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"
Oct 30 03:28:49.903626 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:49.903538    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/configmap/5b5d25c4-b1d0-493a-8c48-0105b692bd56-web-config\") pod \"dcgm-https-exporter\" (UID: \"5b5d25c4-b1d0-493a-8c48-0105b692bd56\") " pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"
Oct 30 03:28:50.004805 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:50.004737    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/configmap/5b5d25c4-b1d0-493a-8c48-0105b692bd56-web-config\") pod \"dcgm-https-exporter\" (UID: \"5b5d25c4-b1d0-493a-8c48-0105b692bd56\") " pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"
Oct 30 03:28:50.004924 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:50.004847    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5b5d25c4-b1d0-493a-8c48-0105b692bd56-tls\") pod \"dcgm-https-exporter\" (UID: \"5b5d25c4-b1d0-493a-8c48-0105b692bd56\") " pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"
Oct 30 03:28:50.004924 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:50.004872    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5b5d25c4-b1d0-493a-8c48-0105b692bd56-data\") pod \"dcgm-https-exporter\" (UID: \"5b5d25c4-b1d0-493a-8c48-0105b692bd56\") " pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"
Oct 30 03:28:50.005225 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:50.005206    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5b5d25c4-b1d0-493a-8c48-0105b692bd56-data\") pod \"dcgm-https-exporter\" (UID: \"5b5d25c4-b1d0-493a-8c48-0105b692bd56\") " pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"
Oct 30 03:28:50.005396 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:50.005377    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/configmap/5b5d25c4-b1d0-493a-8c48-0105b692bd56-web-config\") pod \"dcgm-https-exporter\" (UID: \"5b5d25c4-b1d0-493a-8c48-0105b692bd56\") " pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"
Oct 30 03:28:50.007372 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:50.007355    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5b5d25c4-b1d0-493a-8c48-0105b692bd56-tls\") pod \"dcgm-https-exporter\" (UID: \"5b5d25c4-b1d0-493a-8c48-0105b692bd56\") " pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"
Oct 30 03:28:50.082522 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:50.082500    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"
Oct 30 03:28:50.375434 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:50.375410    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"]
Oct 30 03:28:50.376856 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:28:50.376804    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5d25c4_b1d0_493a_8c48_0105b692bd56.slice/crio-0ef205acf17def599cbc60c823530cb1144ce882f07b0cec057d7ff66e7d856f WatchSource:0}: Error finding container 0ef205acf17def599cbc60c823530cb1144ce882f07b0cec057d7ff66e7d856f: Status 404 returned error can't find the container with id 0ef205acf17def599cbc60c823530cb1144ce882f07b0cec057d7ff66e7d856f
Oct 30 03:28:50.378564 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:50.378548    4960 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:28:50.783444 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:50.783363    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter" event={"ID":"5b5d25c4-b1d0-493a-8c48-0105b692bd56","Type":"ContainerStarted","Data":"0ef205acf17def599cbc60c823530cb1144ce882f07b0cec057d7ff66e7d856f"}
Oct 30 03:28:53.955245 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:28:53.955170    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:28:54.807414 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:54.807375    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter" event={"ID":"5b5d25c4-b1d0-493a-8c48-0105b692bd56","Type":"ContainerStarted","Data":"91f7e71528725f312af7c326e5b33d7e11693ce9eba8e1dfa61d9bc9ee4d94fe"}
Oct 30 03:28:54.807612 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:54.807516    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"
Oct 30 03:28:54.848837 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:54.848776    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter" podStartSLOduration=2.257957221 podStartE2EDuration="5.848762517s" totalImagesPullingTime="3.590805296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:54.847858358 +0000 UTC m=+2815.496906780" watchObservedRunningTime="2026-10-30 03:28:54.848762517 +0000 UTC m=+2815.497810938"
Oct 30 03:28:54.850753 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:28:54.850713    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:00.814641 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:00.814598    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"
Oct 30 03:29:04.528423 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:04.528387    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:04.637662 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:04.637632    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:04.839393 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:04.839369    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:04.958690 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:04.958666    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:06.471025 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:06.470989    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:06.606756 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:06.606713    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:06.722069 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:06.722003    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:08.524830 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:08.524798    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:10.358868 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:10.358839    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:29:10.359347 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:10.359038    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="21894725-c9c8-40c6-9c65-4a6724b9c80f" containerName="otc-container" containerID="cri-o://6cac4c21ff7feec9e757cd8d4ba42f90d24219b1d0ac75bf6028cdce8add7214" gracePeriod=30
Oct 30 03:29:13.414511 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.414483    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:13.497775 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.497696    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21894725-c9c8-40c6-9c65-4a6724b9c80f-kube-api-access-4x5gk\" (UniqueName: \"kubernetes.io/projected/21894725-c9c8-40c6-9c65-4a6724b9c80f-kube-api-access-4x5gk\") pod \"21894725-c9c8-40c6-9c65-4a6724b9c80f\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") "
Oct 30 03:29:13.497775 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.497761    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-otlp-tls-certs\") pod \"21894725-c9c8-40c6-9c65-4a6724b9c80f\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") "
Oct 30 03:29:13.497998 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.497790    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-tls-certs\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-tls-certs\") pod \"21894725-c9c8-40c6-9c65-4a6724b9c80f\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") "
Oct 30 03:29:13.497998 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.497822    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-monitoring-tls-certs\") pod \"21894725-c9c8-40c6-9c65-4a6724b9c80f\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") "
Oct 30 03:29:13.497998 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.497850    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-data-science-collector-ta-client-cert\") pod \"21894725-c9c8-40c6-9c65-4a6724b9c80f\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") "
Oct 30 03:29:13.497998 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.497908    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21894725-c9c8-40c6-9c65-4a6724b9c80f-otc-internal\" (UniqueName: \"kubernetes.io/configmap/21894725-c9c8-40c6-9c65-4a6724b9c80f-otc-internal\") pod \"21894725-c9c8-40c6-9c65-4a6724b9c80f\" (UID: \"21894725-c9c8-40c6-9c65-4a6724b9c80f\") "
Oct 30 03:29:13.498544 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.498474    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21894725-c9c8-40c6-9c65-4a6724b9c80f-otc-internal" pod "21894725-c9c8-40c6-9c65-4a6724b9c80f" (UID: "21894725-c9c8-40c6-9c65-4a6724b9c80f"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:13.500227 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.500146    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-data-science-collector-ta-client-cert" pod "21894725-c9c8-40c6-9c65-4a6724b9c80f" (UID: "21894725-c9c8-40c6-9c65-4a6724b9c80f"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:13.500597 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.500570    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-tls-certs" pod "21894725-c9c8-40c6-9c65-4a6724b9c80f" (UID: "21894725-c9c8-40c6-9c65-4a6724b9c80f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:13.500597 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.500585    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21894725-c9c8-40c6-9c65-4a6724b9c80f-kube-api-access-4x5gk" pod "21894725-c9c8-40c6-9c65-4a6724b9c80f" (UID: "21894725-c9c8-40c6-9c65-4a6724b9c80f"). InnerVolumeSpecName "kube-api-access-4x5gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:13.500735 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.500705    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-monitoring-tls-certs" pod "21894725-c9c8-40c6-9c65-4a6724b9c80f" (UID: "21894725-c9c8-40c6-9c65-4a6724b9c80f"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:13.500772 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.500750    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-otlp-tls-certs" pod "21894725-c9c8-40c6-9c65-4a6724b9c80f" (UID: "21894725-c9c8-40c6-9c65-4a6724b9c80f"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:13.598855 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.598828    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/21894725-c9c8-40c6-9c65-4a6724b9c80f-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:29:13.599001 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.598983    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x5gk\" (UniqueName: \"kubernetes.io/projected/21894725-c9c8-40c6-9c65-4a6724b9c80f-kube-api-access-4x5gk\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:29:13.599073 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.599006    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:29:13.599073 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.599020    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:29:13.599073 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.599032    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:29:13.599073 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.599046    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/21894725-c9c8-40c6-9c65-4a6724b9c80f-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:29:13.896225 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.896170    4960 generic.go:356] "Generic (PLEG): container finished" podID="21894725-c9c8-40c6-9c65-4a6724b9c80f" containerID="6cac4c21ff7feec9e757cd8d4ba42f90d24219b1d0ac75bf6028cdce8add7214" exitCode=0
Oct 30 03:29:13.896374 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.896250    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:13.896374 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.896258    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"21894725-c9c8-40c6-9c65-4a6724b9c80f","Type":"ContainerDied","Data":"6cac4c21ff7feec9e757cd8d4ba42f90d24219b1d0ac75bf6028cdce8add7214"}
Oct 30 03:29:13.896374 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.896298    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"21894725-c9c8-40c6-9c65-4a6724b9c80f","Type":"ContainerDied","Data":"cee5a53aa0b8ceadd555484711bc2058424ca4428a1916608a86c84d0e37855e"}
Oct 30 03:29:13.896374 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.896320    4960 scope.go:122] "RemoveContainer" containerID="6cac4c21ff7feec9e757cd8d4ba42f90d24219b1d0ac75bf6028cdce8add7214"
Oct 30 03:29:13.904773 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.904751    4960 scope.go:122] "RemoveContainer" containerID="6cac4c21ff7feec9e757cd8d4ba42f90d24219b1d0ac75bf6028cdce8add7214"
Oct 30 03:29:13.905041 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:29:13.905023    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cac4c21ff7feec9e757cd8d4ba42f90d24219b1d0ac75bf6028cdce8add7214\": container with ID starting with 6cac4c21ff7feec9e757cd8d4ba42f90d24219b1d0ac75bf6028cdce8add7214 not found: ID does not exist" containerID="6cac4c21ff7feec9e757cd8d4ba42f90d24219b1d0ac75bf6028cdce8add7214"
Oct 30 03:29:13.905088 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.905049    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cac4c21ff7feec9e757cd8d4ba42f90d24219b1d0ac75bf6028cdce8add7214"} err="failed to get container status \"6cac4c21ff7feec9e757cd8d4ba42f90d24219b1d0ac75bf6028cdce8add7214\": rpc error: code = NotFound desc = could not find container \"6cac4c21ff7feec9e757cd8d4ba42f90d24219b1d0ac75bf6028cdce8add7214\": container with ID starting with 6cac4c21ff7feec9e757cd8d4ba42f90d24219b1d0ac75bf6028cdce8add7214 not found: ID does not exist"
Oct 30 03:29:13.949994 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.949972    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:29:13.963583 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.963559    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:29:13.990320 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.990301    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:29:13.990641 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.990629    4960 cpu_manager.go:414] "containerMap: removing container" podUID="21894725-c9c8-40c6-9c65-4a6724b9c80f" containerName="otc-container"
Oct 30 03:29:13.990685 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.990643    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21894725-c9c8-40c6-9c65-4a6724b9c80f" containerName="otc-container"
Oct 30 03:29:13.990748 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.990739    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="21894725-c9c8-40c6-9c65-4a6724b9c80f" containerName="otc-container"
Oct 30 03:29:13.995146 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:13.995131    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.011844 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.011797    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:29:14.011960 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.011924    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:29:14.012101 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.011973    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:29:14.012235 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.012115    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:29:14.012235 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.012120    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:29:14.012355 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.012261    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-db539ca7\""
Oct 30 03:29:14.013455 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.013435    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:29:14.104038 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.104011    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.104140 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.104052    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.104140 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.104075    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.104250 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.104170    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.104250 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.104215    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx5ks\" (UniqueName: \"kubernetes.io/projected/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-kube-api-access-sx5ks\") pod \"data-science-collector-collector-0\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.104250 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.104249    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.205492 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.205437    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.205492 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.205481    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.205643 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.205510    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.205643 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.205532    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.205643 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.205556    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.205643 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.205570    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx5ks\" (UniqueName: \"kubernetes.io/projected/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-kube-api-access-sx5ks\") pod \"data-science-collector-collector-0\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.206461 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.206437    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.208067 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.208040    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.208153 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.208073    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.208471 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.208454    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.208471 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.208463    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.226411 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.226390    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx5ks\" (UniqueName: \"kubernetes.io/projected/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-kube-api-access-sx5ks\") pod \"data-science-collector-collector-0\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.307353 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.307329    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:14.600097 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.600075    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:29:14.601769 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:29:14.601745    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e543a0e_ea16_4a53_8cb8_a99855ff5a92.slice/crio-ea36ebc66923ac618cfc2fa5005ad4d65e828acc08a7c79e516b0f2c2308af7c WatchSource:0}: Error finding container ea36ebc66923ac618cfc2fa5005ad4d65e828acc08a7c79e516b0f2c2308af7c: Status 404 returned error can't find the container with id ea36ebc66923ac618cfc2fa5005ad4d65e828acc08a7c79e516b0f2c2308af7c
Oct 30 03:29:14.604025 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.603990    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:29:14.604093 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.604063    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:29:14.859336 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.859255    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:14.904965 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.904930    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5e543a0e-ea16-4a53-8cb8-a99855ff5a92","Type":"ContainerStarted","Data":"5b4bf796cb1dd4633f2d3524ce2ebb772703e2df6ad2d672938a10735300842a"}
Oct 30 03:29:14.905114 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.904980    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5e543a0e-ea16-4a53-8cb8-a99855ff5a92","Type":"ContainerStarted","Data":"ea36ebc66923ac618cfc2fa5005ad4d65e828acc08a7c79e516b0f2c2308af7c"}
Oct 30 03:29:14.949529 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:14.949477    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.949463623 podStartE2EDuration="1.949463623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:14.949119498 +0000 UTC m=+2835.598167920" watchObservedRunningTime="2026-10-30 03:29:14.949463623 +0000 UTC m=+2835.598512044"
Oct 30 03:29:15.503747 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:15.503710    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21894725-c9c8-40c6-9c65-4a6724b9c80f" path="/var/lib/kubelet/pods/21894725-c9c8-40c6-9c65-4a6724b9c80f/volumes"
Oct 30 03:29:23.955355 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:29:23.955315    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:29:53.953331 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:29:53.953292    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:29:54.958313 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:29:54.958282    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:23.125717 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:30:23.125673    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:23.958352 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:30:23.958307    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:30:53.952858 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:30:53.952819    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:30:55.076266 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:30:55.076235    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:23.953725 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:31:23.953691    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:31:26.423001 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:26.422970    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:27.516449 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:27.516421    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:28.443141 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:28.443108    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:28.589034 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:28.588997    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:28.720313 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:28.720248    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:30.212139 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:30.212102    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:32.064743 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:32.064713    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:31:32.065224 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:32.064938    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="5e543a0e-ea16-4a53-8cb8-a99855ff5a92" containerName="otc-container" containerID="cri-o://5b4bf796cb1dd4633f2d3524ce2ebb772703e2df6ad2d672938a10735300842a" gracePeriod=30
Oct 30 03:31:34.310705 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.310675    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.363353 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.363323    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-tls-certs\") pod \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") "
Oct 30 03:31:34.363499 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.363375    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-data-science-collector-ta-client-cert\") pod \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") "
Oct 30 03:31:34.363499 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.363418    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-otlp-tls-certs\") pod \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") "
Oct 30 03:31:34.363499 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.363463    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-otc-internal\") pod \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") "
Oct 30 03:31:34.363499 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.363482    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-kube-api-access-sx5ks\" (UniqueName: \"kubernetes.io/projected/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-kube-api-access-sx5ks\") pod \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") "
Oct 30 03:31:34.363730 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.363542    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-monitoring-tls-certs\") pod \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\" (UID: \"5e543a0e-ea16-4a53-8cb8-a99855ff5a92\") "
Oct 30 03:31:34.363876 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.363850    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-otc-internal" pod "5e543a0e-ea16-4a53-8cb8-a99855ff5a92" (UID: "5e543a0e-ea16-4a53-8cb8-a99855ff5a92"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:34.367285 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.367256    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-tls-certs" pod "5e543a0e-ea16-4a53-8cb8-a99855ff5a92" (UID: "5e543a0e-ea16-4a53-8cb8-a99855ff5a92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:34.367748 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.367721    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-kube-api-access-sx5ks" pod "5e543a0e-ea16-4a53-8cb8-a99855ff5a92" (UID: "5e543a0e-ea16-4a53-8cb8-a99855ff5a92"). InnerVolumeSpecName "kube-api-access-sx5ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:34.368237 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.367721    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-otlp-tls-certs" pod "5e543a0e-ea16-4a53-8cb8-a99855ff5a92" (UID: "5e543a0e-ea16-4a53-8cb8-a99855ff5a92"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:34.368237 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.368122    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-data-science-collector-ta-client-cert" pod "5e543a0e-ea16-4a53-8cb8-a99855ff5a92" (UID: "5e543a0e-ea16-4a53-8cb8-a99855ff5a92"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:34.371341 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.371310    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-monitoring-tls-certs" pod "5e543a0e-ea16-4a53-8cb8-a99855ff5a92" (UID: "5e543a0e-ea16-4a53-8cb8-a99855ff5a92"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:34.464886 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.464821    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:31:34.464886 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.464848    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:31:34.464886 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.464862    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:31:34.464886 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.464875    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:31:34.465084 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.464888    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx5ks\" (UniqueName: \"kubernetes.io/projected/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-kube-api-access-sx5ks\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:31:34.465084 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.464902    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e543a0e-ea16-4a53-8cb8-a99855ff5a92-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:31:34.579230 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.579197    4960 generic.go:356] "Generic (PLEG): container finished" podID="5e543a0e-ea16-4a53-8cb8-a99855ff5a92" containerID="5b4bf796cb1dd4633f2d3524ce2ebb772703e2df6ad2d672938a10735300842a" exitCode=0
Oct 30 03:31:34.579369 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.579247    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.579369 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.579284    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5e543a0e-ea16-4a53-8cb8-a99855ff5a92","Type":"ContainerDied","Data":"5b4bf796cb1dd4633f2d3524ce2ebb772703e2df6ad2d672938a10735300842a"}
Oct 30 03:31:34.579369 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.579347    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5e543a0e-ea16-4a53-8cb8-a99855ff5a92","Type":"ContainerDied","Data":"ea36ebc66923ac618cfc2fa5005ad4d65e828acc08a7c79e516b0f2c2308af7c"}
Oct 30 03:31:34.579535 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.579375    4960 scope.go:122] "RemoveContainer" containerID="5b4bf796cb1dd4633f2d3524ce2ebb772703e2df6ad2d672938a10735300842a"
Oct 30 03:31:34.588401 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.588378    4960 scope.go:122] "RemoveContainer" containerID="5b4bf796cb1dd4633f2d3524ce2ebb772703e2df6ad2d672938a10735300842a"
Oct 30 03:31:34.588651 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:31:34.588629    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b4bf796cb1dd4633f2d3524ce2ebb772703e2df6ad2d672938a10735300842a\": container with ID starting with 5b4bf796cb1dd4633f2d3524ce2ebb772703e2df6ad2d672938a10735300842a not found: ID does not exist" containerID="5b4bf796cb1dd4633f2d3524ce2ebb772703e2df6ad2d672938a10735300842a"
Oct 30 03:31:34.588712 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.588664    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b4bf796cb1dd4633f2d3524ce2ebb772703e2df6ad2d672938a10735300842a"} err="failed to get container status \"5b4bf796cb1dd4633f2d3524ce2ebb772703e2df6ad2d672938a10735300842a\": rpc error: code = NotFound desc = could not find container \"5b4bf796cb1dd4633f2d3524ce2ebb772703e2df6ad2d672938a10735300842a\": container with ID starting with 5b4bf796cb1dd4633f2d3524ce2ebb772703e2df6ad2d672938a10735300842a not found: ID does not exist"
Oct 30 03:31:34.632427 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.632408    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:31:34.645815 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.645790    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:31:34.669402 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.669379    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:31:34.669751 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.669739    4960 cpu_manager.go:414] "containerMap: removing container" podUID="5e543a0e-ea16-4a53-8cb8-a99855ff5a92" containerName="otc-container"
Oct 30 03:31:34.669795 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.669753    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e543a0e-ea16-4a53-8cb8-a99855ff5a92" containerName="otc-container"
Oct 30 03:31:34.669829 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.669817    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e543a0e-ea16-4a53-8cb8-a99855ff5a92" containerName="otc-container"
Oct 30 03:31:34.674139 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.674123    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.692705 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.692642    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:31:34.692705 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.692669    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:31:34.692873 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.692704    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:31:34.692873 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.692735    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:31:34.692873 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.692800    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:31:34.693084 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.693062    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:31:34.694221 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.694200    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:31:34.767294 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.767220    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.767408 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.767299    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.767408 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.767373    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.767515 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.767422    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/94628639-fc0e-47e7-8fe9-c5df494a00a4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.767515 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.767450    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.767650 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.767511    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfmft\" (UniqueName: \"kubernetes.io/projected/94628639-fc0e-47e7-8fe9-c5df494a00a4-kube-api-access-lfmft\") pod \"data-science-collector-collector-0\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.868774 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.868749    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/94628639-fc0e-47e7-8fe9-c5df494a00a4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.868880 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.868781    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.868880 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.868801    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfmft\" (UniqueName: \"kubernetes.io/projected/94628639-fc0e-47e7-8fe9-c5df494a00a4-kube-api-access-lfmft\") pod \"data-science-collector-collector-0\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.868880 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.868820    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.868880 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.868855    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.868880 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.868879    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.869558 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.869533    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/94628639-fc0e-47e7-8fe9-c5df494a00a4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.871563 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.871544    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.871646 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.871609    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.871731 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.871713    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.871769 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.871715    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.890039 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.890016    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfmft\" (UniqueName: \"kubernetes.io/projected/94628639-fc0e-47e7-8fe9-c5df494a00a4-kube-api-access-lfmft\") pod \"data-science-collector-collector-0\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:34.985916 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:34.985892    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:35.145439 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.145411    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"]
Oct 30 03:31:35.145631 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.145607    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter" podUID="5b5d25c4-b1d0-493a-8c48-0105b692bd56" containerName="dcgm-https-exporter" containerID="cri-o://91f7e71528725f312af7c326e5b33d7e11693ce9eba8e1dfa61d9bc9ee4d94fe" gracePeriod=30
Oct 30 03:31:35.241443 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.241401    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:35.282452 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.282428    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:31:35.283086 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:31:35.283049    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94628639_fc0e_47e7_8fe9_c5df494a00a4.slice/crio-797349b3d5f06de898a1f262bdf70b55b20c7c4bd696cef5f33cd9299dd1d3a6 WatchSource:0}: Error finding container 797349b3d5f06de898a1f262bdf70b55b20c7c4bd696cef5f33cd9299dd1d3a6: Status 404 returned error can't find the container with id 797349b3d5f06de898a1f262bdf70b55b20c7c4bd696cef5f33cd9299dd1d3a6
Oct 30 03:31:35.285849 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.285813    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:31:35.285931 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.285891    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:31:35.403614 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.403590    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"
Oct 30 03:31:35.474588 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.474557    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b5d25c4-b1d0-493a-8c48-0105b692bd56-tls\" (UniqueName: \"kubernetes.io/secret/5b5d25c4-b1d0-493a-8c48-0105b692bd56-tls\") pod \"5b5d25c4-b1d0-493a-8c48-0105b692bd56\" (UID: \"5b5d25c4-b1d0-493a-8c48-0105b692bd56\") "
Oct 30 03:31:35.474766 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.474607    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b5d25c4-b1d0-493a-8c48-0105b692bd56-web-config\" (UniqueName: \"kubernetes.io/configmap/5b5d25c4-b1d0-493a-8c48-0105b692bd56-web-config\") pod \"5b5d25c4-b1d0-493a-8c48-0105b692bd56\" (UID: \"5b5d25c4-b1d0-493a-8c48-0105b692bd56\") "
Oct 30 03:31:35.474766 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.474657    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b5d25c4-b1d0-493a-8c48-0105b692bd56-data\" (UniqueName: \"kubernetes.io/empty-dir/5b5d25c4-b1d0-493a-8c48-0105b692bd56-data\") pod \"5b5d25c4-b1d0-493a-8c48-0105b692bd56\" (UID: \"5b5d25c4-b1d0-493a-8c48-0105b692bd56\") "
Oct 30 03:31:35.475229 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.475202    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b5d25c4-b1d0-493a-8c48-0105b692bd56-data" pod "5b5d25c4-b1d0-493a-8c48-0105b692bd56" (UID: "5b5d25c4-b1d0-493a-8c48-0105b692bd56"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:35.477525 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.477500    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b5d25c4-b1d0-493a-8c48-0105b692bd56-tls" pod "5b5d25c4-b1d0-493a-8c48-0105b692bd56" (UID: "5b5d25c4-b1d0-493a-8c48-0105b692bd56"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:35.480258 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.480238    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b5d25c4-b1d0-493a-8c48-0105b692bd56-web-config" pod "5b5d25c4-b1d0-493a-8c48-0105b692bd56" (UID: "5b5d25c4-b1d0-493a-8c48-0105b692bd56"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:35.506656 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.506628    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e543a0e-ea16-4a53-8cb8-a99855ff5a92" path="/var/lib/kubelet/pods/5e543a0e-ea16-4a53-8cb8-a99855ff5a92/volumes"
Oct 30 03:31:35.575917 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.575887    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5b5d25c4-b1d0-493a-8c48-0105b692bd56-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:31:35.575917 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.575918    4960 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/configmap/5b5d25c4-b1d0-493a-8c48-0105b692bd56-web-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:31:35.576102 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.575933    4960 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5b5d25c4-b1d0-493a-8c48-0105b692bd56-data\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:31:35.586894 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.586863    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"94628639-fc0e-47e7-8fe9-c5df494a00a4","Type":"ContainerStarted","Data":"59f8e32f79c03e948c3d8acfb045fb16571200ef13ca9e075acc0ac3a601b058"}
Oct 30 03:31:35.587037 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.586934    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"94628639-fc0e-47e7-8fe9-c5df494a00a4","Type":"ContainerStarted","Data":"797349b3d5f06de898a1f262bdf70b55b20c7c4bd696cef5f33cd9299dd1d3a6"}
Oct 30 03:31:35.588114 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.588093    4960 generic.go:356] "Generic (PLEG): container finished" podID="5b5d25c4-b1d0-493a-8c48-0105b692bd56" containerID="91f7e71528725f312af7c326e5b33d7e11693ce9eba8e1dfa61d9bc9ee4d94fe" exitCode=0
Oct 30 03:31:35.588263 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.588135    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"
Oct 30 03:31:35.588263 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.588154    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter" event={"ID":"5b5d25c4-b1d0-493a-8c48-0105b692bd56","Type":"ContainerDied","Data":"91f7e71528725f312af7c326e5b33d7e11693ce9eba8e1dfa61d9bc9ee4d94fe"}
Oct 30 03:31:35.588263 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.588206    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter" event={"ID":"5b5d25c4-b1d0-493a-8c48-0105b692bd56","Type":"ContainerDied","Data":"0ef205acf17def599cbc60c823530cb1144ce882f07b0cec057d7ff66e7d856f"}
Oct 30 03:31:35.588263 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.588228    4960 scope.go:122] "RemoveContainer" containerID="91f7e71528725f312af7c326e5b33d7e11693ce9eba8e1dfa61d9bc9ee4d94fe"
Oct 30 03:31:35.596718 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.596695    4960 scope.go:122] "RemoveContainer" containerID="91f7e71528725f312af7c326e5b33d7e11693ce9eba8e1dfa61d9bc9ee4d94fe"
Oct 30 03:31:35.597048 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:31:35.597001    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91f7e71528725f312af7c326e5b33d7e11693ce9eba8e1dfa61d9bc9ee4d94fe\": container with ID starting with 91f7e71528725f312af7c326e5b33d7e11693ce9eba8e1dfa61d9bc9ee4d94fe not found: ID does not exist" containerID="91f7e71528725f312af7c326e5b33d7e11693ce9eba8e1dfa61d9bc9ee4d94fe"
Oct 30 03:31:35.597117 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.597056    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f7e71528725f312af7c326e5b33d7e11693ce9eba8e1dfa61d9bc9ee4d94fe"} err="failed to get container status \"91f7e71528725f312af7c326e5b33d7e11693ce9eba8e1dfa61d9bc9ee4d94fe\": rpc error: code = NotFound desc = could not find container \"91f7e71528725f312af7c326e5b33d7e11693ce9eba8e1dfa61d9bc9ee4d94fe\": container with ID starting with 91f7e71528725f312af7c326e5b33d7e11693ce9eba8e1dfa61d9bc9ee4d94fe not found: ID does not exist"
Oct 30 03:31:35.630406 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.630369    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.630356716 podStartE2EDuration="1.630356716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:31:35.629394671 +0000 UTC m=+2976.278443093" watchObservedRunningTime="2026-10-30 03:31:35.630356716 +0000 UTC m=+2976.279405156"
Oct 30 03:31:35.666568 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.666540    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"]
Oct 30 03:31:35.668416 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:35.668395    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["tests-monitoring-dcgm-https-18e330fa9f8b0f51/dcgm-https-exporter"]
Oct 30 03:31:37.503593 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:37.503554    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b5d25c4-b1d0-493a-8c48-0105b692bd56" path="/var/lib/kubelet/pods/5b5d25c4-b1d0-493a-8c48-0105b692bd56/volumes"
Oct 30 03:31:40.411302 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:40.411265    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:41.072341 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:41.072304    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:45.629527 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:45.629490    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:45.746419 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:45.746387    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:45.944654 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:45.944574    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:46.066821 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:46.066793    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:47.614546 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:47.614514    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:47.726433 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:47.726401    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:49.159322 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:49.159287    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:31:49.159714 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:49.159552    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="94628639-fc0e-47e7-8fe9-c5df494a00a4" containerName="otc-container" containerID="cri-o://59f8e32f79c03e948c3d8acfb045fb16571200ef13ca9e075acc0ac3a601b058" gracePeriod=30
Oct 30 03:31:50.408840 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.408811    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:50.490030 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.489966    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-tls-certs\") pod \"94628639-fc0e-47e7-8fe9-c5df494a00a4\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") "
Oct 30 03:31:50.490030 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.490003    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94628639-fc0e-47e7-8fe9-c5df494a00a4-kube-api-access-lfmft\" (UniqueName: \"kubernetes.io/projected/94628639-fc0e-47e7-8fe9-c5df494a00a4-kube-api-access-lfmft\") pod \"94628639-fc0e-47e7-8fe9-c5df494a00a4\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") "
Oct 30 03:31:50.490286 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.490043    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94628639-fc0e-47e7-8fe9-c5df494a00a4-otc-internal\" (UniqueName: \"kubernetes.io/configmap/94628639-fc0e-47e7-8fe9-c5df494a00a4-otc-internal\") pod \"94628639-fc0e-47e7-8fe9-c5df494a00a4\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") "
Oct 30 03:31:50.490286 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.490068    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-otlp-tls-certs\") pod \"94628639-fc0e-47e7-8fe9-c5df494a00a4\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") "
Oct 30 03:31:50.490286 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.490144    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-data-science-collector-ta-client-cert\") pod \"94628639-fc0e-47e7-8fe9-c5df494a00a4\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") "
Oct 30 03:31:50.490286 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.490203    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-monitoring-tls-certs\") pod \"94628639-fc0e-47e7-8fe9-c5df494a00a4\" (UID: \"94628639-fc0e-47e7-8fe9-c5df494a00a4\") "
Oct 30 03:31:50.490496 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.490472    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94628639-fc0e-47e7-8fe9-c5df494a00a4-otc-internal" pod "94628639-fc0e-47e7-8fe9-c5df494a00a4" (UID: "94628639-fc0e-47e7-8fe9-c5df494a00a4"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:50.492423 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.492394    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94628639-fc0e-47e7-8fe9-c5df494a00a4-kube-api-access-lfmft" pod "94628639-fc0e-47e7-8fe9-c5df494a00a4" (UID: "94628639-fc0e-47e7-8fe9-c5df494a00a4"). InnerVolumeSpecName "kube-api-access-lfmft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:50.492558 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.492535    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-data-science-collector-ta-client-cert" pod "94628639-fc0e-47e7-8fe9-c5df494a00a4" (UID: "94628639-fc0e-47e7-8fe9-c5df494a00a4"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:50.492617 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.492591    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-monitoring-tls-certs" pod "94628639-fc0e-47e7-8fe9-c5df494a00a4" (UID: "94628639-fc0e-47e7-8fe9-c5df494a00a4"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:50.492825 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.492803    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-tls-certs" pod "94628639-fc0e-47e7-8fe9-c5df494a00a4" (UID: "94628639-fc0e-47e7-8fe9-c5df494a00a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:50.492918 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.492832    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-otlp-tls-certs" pod "94628639-fc0e-47e7-8fe9-c5df494a00a4" (UID: "94628639-fc0e-47e7-8fe9-c5df494a00a4"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:50.591347 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.591324    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:31:50.591347 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.591346    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:31:50.591478 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.591357    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:31:50.591478 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.591366    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfmft\" (UniqueName: \"kubernetes.io/projected/94628639-fc0e-47e7-8fe9-c5df494a00a4-kube-api-access-lfmft\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:31:50.591478 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.591375    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/94628639-fc0e-47e7-8fe9-c5df494a00a4-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:31:50.591478 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.591383    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/94628639-fc0e-47e7-8fe9-c5df494a00a4-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:31:50.670884 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.670862    4960 generic.go:356] "Generic (PLEG): container finished" podID="94628639-fc0e-47e7-8fe9-c5df494a00a4" containerID="59f8e32f79c03e948c3d8acfb045fb16571200ef13ca9e075acc0ac3a601b058" exitCode=0
Oct 30 03:31:50.670993 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.670907    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:50.670993 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.670956    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"94628639-fc0e-47e7-8fe9-c5df494a00a4","Type":"ContainerDied","Data":"59f8e32f79c03e948c3d8acfb045fb16571200ef13ca9e075acc0ac3a601b058"}
Oct 30 03:31:50.671123 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.671015    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"94628639-fc0e-47e7-8fe9-c5df494a00a4","Type":"ContainerDied","Data":"797349b3d5f06de898a1f262bdf70b55b20c7c4bd696cef5f33cd9299dd1d3a6"}
Oct 30 03:31:50.671123 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.671049    4960 scope.go:122] "RemoveContainer" containerID="59f8e32f79c03e948c3d8acfb045fb16571200ef13ca9e075acc0ac3a601b058"
Oct 30 03:31:50.680542 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.680517    4960 scope.go:122] "RemoveContainer" containerID="59f8e32f79c03e948c3d8acfb045fb16571200ef13ca9e075acc0ac3a601b058"
Oct 30 03:31:50.680795 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:31:50.680779    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f8e32f79c03e948c3d8acfb045fb16571200ef13ca9e075acc0ac3a601b058\": container with ID starting with 59f8e32f79c03e948c3d8acfb045fb16571200ef13ca9e075acc0ac3a601b058 not found: ID does not exist" containerID="59f8e32f79c03e948c3d8acfb045fb16571200ef13ca9e075acc0ac3a601b058"
Oct 30 03:31:50.680844 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.680802    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f8e32f79c03e948c3d8acfb045fb16571200ef13ca9e075acc0ac3a601b058"} err="failed to get container status \"59f8e32f79c03e948c3d8acfb045fb16571200ef13ca9e075acc0ac3a601b058\": rpc error: code = NotFound desc = could not find container \"59f8e32f79c03e948c3d8acfb045fb16571200ef13ca9e075acc0ac3a601b058\": container with ID starting with 59f8e32f79c03e948c3d8acfb045fb16571200ef13ca9e075acc0ac3a601b058 not found: ID does not exist"
Oct 30 03:31:50.725733 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.725710    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:31:50.739652 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.739629    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:31:50.774054 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.774006    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:31:50.774399 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.774385    4960 cpu_manager.go:414] "containerMap: removing container" podUID="5b5d25c4-b1d0-493a-8c48-0105b692bd56" containerName="dcgm-https-exporter"
Oct 30 03:31:50.774451 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.774401    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b5d25c4-b1d0-493a-8c48-0105b692bd56" containerName="dcgm-https-exporter"
Oct 30 03:31:50.774451 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.774412    4960 cpu_manager.go:414] "containerMap: removing container" podUID="94628639-fc0e-47e7-8fe9-c5df494a00a4" containerName="otc-container"
Oct 30 03:31:50.774451 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.774418    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94628639-fc0e-47e7-8fe9-c5df494a00a4" containerName="otc-container"
Oct 30 03:31:50.774539 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.774477    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b5d25c4-b1d0-493a-8c48-0105b692bd56" containerName="dcgm-https-exporter"
Oct 30 03:31:50.774539 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.774485    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="94628639-fc0e-47e7-8fe9-c5df494a00a4" containerName="otc-container"
Oct 30 03:31:50.779319 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.779300    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:50.798535 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.798514    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:31:50.798616 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.798604    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:31:50.798673 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.798626    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:31:50.798673 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.798650    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:31:50.798673 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.798650    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:31:50.798981 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.798968    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:31:50.810511 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.810491    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:31:50.837353 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.837333    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:50.869700 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.869678    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g"]
Oct 30 03:31:50.873230 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.873212    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g"
Oct 30 03:31:50.891627 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.891605    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g"]
Oct 30 03:31:50.893508 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.893484    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:50.893594 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.893574    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4r6w\" (UniqueName: \"kubernetes.io/projected/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-kube-api-access-p4r6w\") pod \"data-science-collector-collector-0\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:50.893639 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.893617    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:50.893691 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.893665    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:50.893743 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.893727    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:50.893774 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.893755    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:50.896281 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.896250    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"tests-monitoring-tenant-metrics\"/\"default-dockercfg-wq7h5\""
Oct 30 03:31:50.898750 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.898733    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-tenant-metrics\"/\"openshift-service-ca.crt\""
Oct 30 03:31:50.898832 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.898819    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-tenant-metrics\"/\"kube-root-ca.crt\""
Oct 30 03:31:50.970077 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.970042    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:50.994453 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.994425    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3e40-a069-4855-9694-1dabf09907b2-data\") pod \"tenant-metrics-exporter-5cf544fb97-jfb9g\" (UID: \"a3fd3e40-a069-4855-9694-1dabf09907b2\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g"
Oct 30 03:31:50.994599 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.994462    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:50.994599 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.994495    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:50.994599 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.994549    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:50.994599 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.994574    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:50.994830 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.994609    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2phr\" (UniqueName: \"kubernetes.io/projected/a3fd3e40-a069-4855-9694-1dabf09907b2-kube-api-access-s2phr\") pod \"tenant-metrics-exporter-5cf544fb97-jfb9g\" (UID: \"a3fd3e40-a069-4855-9694-1dabf09907b2\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g"
Oct 30 03:31:50.994830 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.994640    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:50.994830 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.994682    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4r6w\" (UniqueName: \"kubernetes.io/projected/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-kube-api-access-p4r6w\") pod \"data-science-collector-collector-0\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:50.995303 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.995276    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:50.997256 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.997232    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:50.997425 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.997404    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:50.997514 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.997495    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:50.997864 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:50.997848    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:51.015167 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.015144    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4r6w\" (UniqueName: \"kubernetes.io/projected/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-kube-api-access-p4r6w\") pod \"data-science-collector-collector-0\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:51.091698 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.091676    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:31:51.095962 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.095936    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2phr\" (UniqueName: \"kubernetes.io/projected/a3fd3e40-a069-4855-9694-1dabf09907b2-kube-api-access-s2phr\") pod \"tenant-metrics-exporter-5cf544fb97-jfb9g\" (UID: \"a3fd3e40-a069-4855-9694-1dabf09907b2\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g"
Oct 30 03:31:51.096070 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.096011    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3e40-a069-4855-9694-1dabf09907b2-data\") pod \"tenant-metrics-exporter-5cf544fb97-jfb9g\" (UID: \"a3fd3e40-a069-4855-9694-1dabf09907b2\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g"
Oct 30 03:31:51.096505 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.096483    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3e40-a069-4855-9694-1dabf09907b2-data\") pod \"tenant-metrics-exporter-5cf544fb97-jfb9g\" (UID: \"a3fd3e40-a069-4855-9694-1dabf09907b2\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g"
Oct 30 03:31:51.119811 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.119788    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2phr\" (UniqueName: \"kubernetes.io/projected/a3fd3e40-a069-4855-9694-1dabf09907b2-kube-api-access-s2phr\") pod \"tenant-metrics-exporter-5cf544fb97-jfb9g\" (UID: \"a3fd3e40-a069-4855-9694-1dabf09907b2\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g"
Oct 30 03:31:51.177572 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.177544    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:51.187580 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.187558    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g"
Oct 30 03:31:51.300993 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.300967    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:51.393052 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.393030    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:31:51.395025 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:31:51.394995    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b91a47_8c1e_4a9a_b3e8_9ec55580b676.slice/crio-cdfd996e299800ebb370f47443f65d4ac0873d7648a4c5a146fda65058b50e0d WatchSource:0}: Error finding container cdfd996e299800ebb370f47443f65d4ac0873d7648a4c5a146fda65058b50e0d: Status 404 returned error can't find the container with id cdfd996e299800ebb370f47443f65d4ac0873d7648a4c5a146fda65058b50e0d
Oct 30 03:31:51.397277 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.397242    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:31:51.397340 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.397298    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:31:51.482461 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.482434    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g"]
Oct 30 03:31:51.483434 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:31:51.483411    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3fd3e40_a069_4855_9694_1dabf09907b2.slice/crio-10fef17b777b98bce46c4b3f4d919197f4b5c4f94c3c1232dab90e2c9778cd20 WatchSource:0}: Error finding container 10fef17b777b98bce46c4b3f4d919197f4b5c4f94c3c1232dab90e2c9778cd20: Status 404 returned error can't find the container with id 10fef17b777b98bce46c4b3f4d919197f4b5c4f94c3c1232dab90e2c9778cd20
Oct 30 03:31:51.503320 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.503296    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94628639-fc0e-47e7-8fe9-c5df494a00a4" path="/var/lib/kubelet/pods/94628639-fc0e-47e7-8fe9-c5df494a00a4/volumes"
Oct 30 03:31:51.678104 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.678023    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g" event={"ID":"a3fd3e40-a069-4855-9694-1dabf09907b2","Type":"ContainerStarted","Data":"5aad7ceec44361baad427fc1e596913a692a6e4426d9e4bd6e5c1b26d7b6b01a"}
Oct 30 03:31:51.678104 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.678088    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g" event={"ID":"a3fd3e40-a069-4855-9694-1dabf09907b2","Type":"ContainerStarted","Data":"10fef17b777b98bce46c4b3f4d919197f4b5c4f94c3c1232dab90e2c9778cd20"}
Oct 30 03:31:51.679891 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.679862    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"58b91a47-8c1e-4a9a-b3e8-9ec55580b676","Type":"ContainerStarted","Data":"ca82d67f460b8f4fe92ce944769eb74db29b049983cf9f8752e8258ceb56e93d"}
Oct 30 03:31:51.680048 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.679909    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"58b91a47-8c1e-4a9a-b3e8-9ec55580b676","Type":"ContainerStarted","Data":"cdfd996e299800ebb370f47443f65d4ac0873d7648a4c5a146fda65058b50e0d"}
Oct 30 03:31:51.720653 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.720607    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g" podStartSLOduration=1.7205960070000001 podStartE2EDuration="1.720596007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:31:51.719653508 +0000 UTC m=+2992.368701929" watchObservedRunningTime="2026-10-30 03:31:51.720596007 +0000 UTC m=+2992.369644425"
Oct 30 03:31:51.769289 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:51.769236    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.769219425 podStartE2EDuration="1.769219425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:31:51.768103646 +0000 UTC m=+2992.417152067" watchObservedRunningTime="2026-10-30 03:31:51.769219425 +0000 UTC m=+2992.418267848"
Oct 30 03:31:53.953117 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:31:53.953085    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:31:55.188343 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:55.188309    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:59.782841 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:59.782808    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:31:59.787539 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:59.787516    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:31:59.788509 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:59.788482    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:31:59.793369 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:31:59.793354    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:32:03.782404 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:32:03.782364    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:03.914015 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:32:03.913980    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:23.952961 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:32:23.952879    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:32:53.953122 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:32:53.953084    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:32:55.325219 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:32:55.325146    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:56.000713 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:32:56.000675    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:23.953236 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:33:23.953197    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:33:44.618645 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:33:44.618608    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:53.954742 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:33:53.954690    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:33:55.417668 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:33:55.417635    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:23.953021 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:34:23.952983    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:34:53.953928 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:34:53.953887    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:34:56.723836 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:34:56.723801    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:35:23.952818 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:35:23.952734    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:35:53.953613 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:35:53.953575    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:35:54.820844 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:35:54.820807    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:35:55.661857 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:35:55.661819    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:00.730963 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:00.730932    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:00.861940 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:00.861907    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:00.888946 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:00.888911    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g"]
Oct 30 03:36:00.889155 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:00.889133    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g" podUID="a3fd3e40-a069-4855-9694-1dabf09907b2" containerName="tenant-metrics-exporter" containerID="cri-o://5aad7ceec44361baad427fc1e596913a692a6e4426d9e4bd6e5c1b26d7b6b01a" gracePeriod=30
Oct 30 03:36:00.986125 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:00.986053    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:01.106740 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:01.106695    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:01.246868 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:01.246812    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g"
Oct 30 03:36:01.414023 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:01.413989    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3fd3e40-a069-4855-9694-1dabf09907b2-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3e40-a069-4855-9694-1dabf09907b2-data\") pod \"a3fd3e40-a069-4855-9694-1dabf09907b2\" (UID: \"a3fd3e40-a069-4855-9694-1dabf09907b2\") "
Oct 30 03:36:01.414172 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:01.414043    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3fd3e40-a069-4855-9694-1dabf09907b2-kube-api-access-s2phr\" (UniqueName: \"kubernetes.io/projected/a3fd3e40-a069-4855-9694-1dabf09907b2-kube-api-access-s2phr\") pod \"a3fd3e40-a069-4855-9694-1dabf09907b2\" (UID: \"a3fd3e40-a069-4855-9694-1dabf09907b2\") "
Oct 30 03:36:01.414396 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:01.414375    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fd3e40-a069-4855-9694-1dabf09907b2-data" pod "a3fd3e40-a069-4855-9694-1dabf09907b2" (UID: "a3fd3e40-a069-4855-9694-1dabf09907b2"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:36:01.416363 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:01.416346    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3fd3e40-a069-4855-9694-1dabf09907b2-kube-api-access-s2phr" pod "a3fd3e40-a069-4855-9694-1dabf09907b2" (UID: "a3fd3e40-a069-4855-9694-1dabf09907b2"). InnerVolumeSpecName "kube-api-access-s2phr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:36:01.515051 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:01.515027    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2phr\" (UniqueName: \"kubernetes.io/projected/a3fd3e40-a069-4855-9694-1dabf09907b2-kube-api-access-s2phr\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:36:01.515051 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:01.515053    4960 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3e40-a069-4855-9694-1dabf09907b2-data\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:36:01.847766 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:01.847733    4960 generic.go:356] "Generic (PLEG): container finished" podID="a3fd3e40-a069-4855-9694-1dabf09907b2" containerID="5aad7ceec44361baad427fc1e596913a692a6e4426d9e4bd6e5c1b26d7b6b01a" exitCode=0
Oct 30 03:36:01.848163 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:01.847781    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g"
Oct 30 03:36:01.848163 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:01.847793    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g" event={"ID":"a3fd3e40-a069-4855-9694-1dabf09907b2","Type":"ContainerDied","Data":"5aad7ceec44361baad427fc1e596913a692a6e4426d9e4bd6e5c1b26d7b6b01a"}
Oct 30 03:36:01.848163 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:01.847831    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g" event={"ID":"a3fd3e40-a069-4855-9694-1dabf09907b2","Type":"ContainerDied","Data":"10fef17b777b98bce46c4b3f4d919197f4b5c4f94c3c1232dab90e2c9778cd20"}
Oct 30 03:36:01.848163 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:01.847852    4960 scope.go:122] "RemoveContainer" containerID="5aad7ceec44361baad427fc1e596913a692a6e4426d9e4bd6e5c1b26d7b6b01a"
Oct 30 03:36:01.858276 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:01.858245    4960 scope.go:122] "RemoveContainer" containerID="5aad7ceec44361baad427fc1e596913a692a6e4426d9e4bd6e5c1b26d7b6b01a"
Oct 30 03:36:01.858637 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:36:01.858611    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aad7ceec44361baad427fc1e596913a692a6e4426d9e4bd6e5c1b26d7b6b01a\": container with ID starting with 5aad7ceec44361baad427fc1e596913a692a6e4426d9e4bd6e5c1b26d7b6b01a not found: ID does not exist" containerID="5aad7ceec44361baad427fc1e596913a692a6e4426d9e4bd6e5c1b26d7b6b01a"
Oct 30 03:36:01.858750 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:01.858642    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aad7ceec44361baad427fc1e596913a692a6e4426d9e4bd6e5c1b26d7b6b01a"} err="failed to get container status \"5aad7ceec44361baad427fc1e596913a692a6e4426d9e4bd6e5c1b26d7b6b01a\": rpc error: code = NotFound desc = could not find container \"5aad7ceec44361baad427fc1e596913a692a6e4426d9e4bd6e5c1b26d7b6b01a\": container with ID starting with 5aad7ceec44361baad427fc1e596913a692a6e4426d9e4bd6e5c1b26d7b6b01a not found: ID does not exist"
Oct 30 03:36:01.901510 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:01.901452    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g"]
Oct 30 03:36:01.905004 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:01.904971    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jfb9g"]
Oct 30 03:36:03.508837 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:03.508790    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3fd3e40-a069-4855-9694-1dabf09907b2" path="/var/lib/kubelet/pods/a3fd3e40-a069-4855-9694-1dabf09907b2/volumes"
Oct 30 03:36:06.633061 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:06.633022    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:06.840014 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:06.839979    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:07.127786 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:07.127757    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:12.933146 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:12.933111    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:13.056710 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:13.056678    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:16.713485 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:16.713453    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:16.833889 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:16.833851    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:16.945200 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:16.945149    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:17.113193 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:17.113142    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:18.742599 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:18.742567    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:18.873355 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:18.873325    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:22.426299 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:22.426264    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:36:22.426797 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:22.426522    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="58b91a47-8c1e-4a9a-b3e8-9ec55580b676" containerName="otc-container" containerID="cri-o://ca82d67f460b8f4fe92ce944769eb74db29b049983cf9f8752e8258ceb56e93d" gracePeriod=30
Oct 30 03:36:23.953692 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:36:23.953657    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:36:29.276201 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.276152    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:29.324314 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.324278    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-otc-internal\" (UniqueName: \"kubernetes.io/configmap/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-otc-internal\") pod \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") "
Oct 30 03:36:29.324314 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.324314    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-monitoring-tls-certs\") pod \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") "
Oct 30 03:36:29.324528 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.324476    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-kube-api-access-p4r6w\" (UniqueName: \"kubernetes.io/projected/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-kube-api-access-p4r6w\") pod \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") "
Oct 30 03:36:29.324528 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.324519    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-tls-certs\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-tls-certs\") pod \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") "
Oct 30 03:36:29.324638 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.324597    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-data-science-collector-ta-client-cert\") pod \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") "
Oct 30 03:36:29.324638 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.324633    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-otlp-tls-certs\") pod \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\" (UID: \"58b91a47-8c1e-4a9a-b3e8-9ec55580b676\") "
Oct 30 03:36:29.324749 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.324683    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-otc-internal" pod "58b91a47-8c1e-4a9a-b3e8-9ec55580b676" (UID: "58b91a47-8c1e-4a9a-b3e8-9ec55580b676"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:36:29.324948 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.324927    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:36:29.327071 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.327020    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-kube-api-access-p4r6w" pod "58b91a47-8c1e-4a9a-b3e8-9ec55580b676" (UID: "58b91a47-8c1e-4a9a-b3e8-9ec55580b676"). InnerVolumeSpecName "kube-api-access-p4r6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:36:29.327761 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.327737    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-monitoring-tls-certs" pod "58b91a47-8c1e-4a9a-b3e8-9ec55580b676" (UID: "58b91a47-8c1e-4a9a-b3e8-9ec55580b676"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:36:29.328056 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.328005    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-tls-certs" pod "58b91a47-8c1e-4a9a-b3e8-9ec55580b676" (UID: "58b91a47-8c1e-4a9a-b3e8-9ec55580b676"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:36:29.328292 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.328270    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-otlp-tls-certs" pod "58b91a47-8c1e-4a9a-b3e8-9ec55580b676" (UID: "58b91a47-8c1e-4a9a-b3e8-9ec55580b676"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:36:29.328366 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.328342    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-data-science-collector-ta-client-cert" pod "58b91a47-8c1e-4a9a-b3e8-9ec55580b676" (UID: "58b91a47-8c1e-4a9a-b3e8-9ec55580b676"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:36:29.425594 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.425569    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4r6w\" (UniqueName: \"kubernetes.io/projected/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-kube-api-access-p4r6w\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:36:29.425594 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.425594    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:36:29.425721 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.425606    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:36:29.425721 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.425615    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:36:29.425721 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.425624    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/58b91a47-8c1e-4a9a-b3e8-9ec55580b676-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:36:29.978431 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.978396    4960 generic.go:356] "Generic (PLEG): container finished" podID="58b91a47-8c1e-4a9a-b3e8-9ec55580b676" containerID="ca82d67f460b8f4fe92ce944769eb74db29b049983cf9f8752e8258ceb56e93d" exitCode=0
Oct 30 03:36:29.978620 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.978435    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"58b91a47-8c1e-4a9a-b3e8-9ec55580b676","Type":"ContainerDied","Data":"ca82d67f460b8f4fe92ce944769eb74db29b049983cf9f8752e8258ceb56e93d"}
Oct 30 03:36:29.978620 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.978484    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"58b91a47-8c1e-4a9a-b3e8-9ec55580b676","Type":"ContainerDied","Data":"cdfd996e299800ebb370f47443f65d4ac0873d7648a4c5a146fda65058b50e0d"}
Oct 30 03:36:29.978620 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.978510    4960 scope.go:122] "RemoveContainer" containerID="ca82d67f460b8f4fe92ce944769eb74db29b049983cf9f8752e8258ceb56e93d"
Oct 30 03:36:29.978620 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.978441    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:29.987330 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.987313    4960 scope.go:122] "RemoveContainer" containerID="ca82d67f460b8f4fe92ce944769eb74db29b049983cf9f8752e8258ceb56e93d"
Oct 30 03:36:29.987601 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:36:29.987577    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca82d67f460b8f4fe92ce944769eb74db29b049983cf9f8752e8258ceb56e93d\": container with ID starting with ca82d67f460b8f4fe92ce944769eb74db29b049983cf9f8752e8258ceb56e93d not found: ID does not exist" containerID="ca82d67f460b8f4fe92ce944769eb74db29b049983cf9f8752e8258ceb56e93d"
Oct 30 03:36:29.987698 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:29.987609    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca82d67f460b8f4fe92ce944769eb74db29b049983cf9f8752e8258ceb56e93d"} err="failed to get container status \"ca82d67f460b8f4fe92ce944769eb74db29b049983cf9f8752e8258ceb56e93d\": rpc error: code = NotFound desc = could not find container \"ca82d67f460b8f4fe92ce944769eb74db29b049983cf9f8752e8258ceb56e93d\": container with ID starting with ca82d67f460b8f4fe92ce944769eb74db29b049983cf9f8752e8258ceb56e93d not found: ID does not exist"
Oct 30 03:36:30.037755 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.037723    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:36:30.051081 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.051055    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:36:30.073791 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.073771    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:36:30.074127 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.074115    4960 cpu_manager.go:414] "containerMap: removing container" podUID="58b91a47-8c1e-4a9a-b3e8-9ec55580b676" containerName="otc-container"
Oct 30 03:36:30.074173 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.074128    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58b91a47-8c1e-4a9a-b3e8-9ec55580b676" containerName="otc-container"
Oct 30 03:36:30.074173 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.074146    4960 cpu_manager.go:414] "containerMap: removing container" podUID="a3fd3e40-a069-4855-9694-1dabf09907b2" containerName="tenant-metrics-exporter"
Oct 30 03:36:30.074173 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.074151    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3fd3e40-a069-4855-9694-1dabf09907b2" containerName="tenant-metrics-exporter"
Oct 30 03:36:30.074299 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.074222    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="58b91a47-8c1e-4a9a-b3e8-9ec55580b676" containerName="otc-container"
Oct 30 03:36:30.074299 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.074232    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3fd3e40-a069-4855-9694-1dabf09907b2" containerName="tenant-metrics-exporter"
Oct 30 03:36:30.078515 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.078499    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.096191 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.096157    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:36:30.108109 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.108085    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:36:30.108109 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.108094    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:36:30.108304 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.108118    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:36:30.108304 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.108148    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:36:30.108304 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.108202    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:36:30.108304 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.108202    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:36:30.131032 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.131010    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.131141 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.131042    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ce464ccc-6523-4de8-9d11-892149d242cd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.131141 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.131072    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj4l4\" (UniqueName: \"kubernetes.io/projected/ce464ccc-6523-4de8-9d11-892149d242cd-kube-api-access-lj4l4\") pod \"data-science-collector-collector-0\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.131289 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.131144    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.131289 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.131211    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.131289 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.131230    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.231598 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.231541    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.231598 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.231574    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ce464ccc-6523-4de8-9d11-892149d242cd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.231743 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.231597    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj4l4\" (UniqueName: \"kubernetes.io/projected/ce464ccc-6523-4de8-9d11-892149d242cd-kube-api-access-lj4l4\") pod \"data-science-collector-collector-0\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.231743 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.231635    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.231743 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.231679    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.231743 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.231708    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.232376 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.232338    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ce464ccc-6523-4de8-9d11-892149d242cd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.234534 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.234507    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.234612 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.234545    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.234761 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.234743    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.234846 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.234827    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.253025 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.253002    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj4l4\" (UniqueName: \"kubernetes.io/projected/ce464ccc-6523-4de8-9d11-892149d242cd-kube-api-access-lj4l4\") pod \"data-science-collector-collector-0\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.390755 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.390729    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:36:30.683228 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.683204    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:36:30.685062 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:36:30.685034    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce464ccc_6523_4de8_9d11_892149d242cd.slice/crio-d35114c87a2f167018fd69225bc048e9d86c94eb9045f5dad29aa3bdbca64bca WatchSource:0}: Error finding container d35114c87a2f167018fd69225bc048e9d86c94eb9045f5dad29aa3bdbca64bca: Status 404 returned error can't find the container with id d35114c87a2f167018fd69225bc048e9d86c94eb9045f5dad29aa3bdbca64bca
Oct 30 03:36:30.687149 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.687115    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:36:30.687255 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.687205    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:36:30.986972 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.986892    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ce464ccc-6523-4de8-9d11-892149d242cd","Type":"ContainerStarted","Data":"07e375d347c3e033ef9ce5b714fa9f4198d5cac840d478c9f062e9d5640fdd26"}
Oct 30 03:36:30.986972 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:30.986954    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ce464ccc-6523-4de8-9d11-892149d242cd","Type":"ContainerStarted","Data":"d35114c87a2f167018fd69225bc048e9d86c94eb9045f5dad29aa3bdbca64bca"}
Oct 30 03:36:31.029717 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:31.029671    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.02965642 podStartE2EDuration="1.02965642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:36:31.028826938 +0000 UTC m=+3271.677875364" watchObservedRunningTime="2026-10-30 03:36:31.02965642 +0000 UTC m=+3271.678704840"
Oct 30 03:36:31.502783 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:31.502750    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58b91a47-8c1e-4a9a-b3e8-9ec55580b676" path="/var/lib/kubelet/pods/58b91a47-8c1e-4a9a-b3e8-9ec55580b676/volumes"
Oct 30 03:36:53.953780 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:36:53.953740    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:36:54.874678 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:54.874632    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:56.002644 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:56.002603    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:57.119781 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:57.119749    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:59.811502 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:59.811469    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:36:59.817238 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:59.817215    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:36:59.817502 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:59.817484    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:36:59.823167 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:36:59.823145    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:37:14.806001 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:37:14.805960    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:23.957274 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:37:23.957224    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:37:53.953339 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:37:53.953302    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:37:55.935639 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:37:55.935606    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:23.953881 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:38:23.953793    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:38:53.954217 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:38:53.954151    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:38:56.026987 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:38:56.026952    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:23.953395 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:39:23.953358    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:39:53.953132 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:39:53.953095    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:39:56.153172 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:39:56.153143    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:11.730128 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:11.730090    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:11.866658 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:11.866624    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:17.713010 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:17.712973    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:17.851699 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:17.851664    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:22.948824 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:22.948785    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:23.115400 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:23.115366    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:23.956315 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:40:23.956270    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:40:27.435772 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:27.435729    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:37.753045 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:37.753007    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:41.155300 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.155266    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"]
Oct 30 03:40:41.155797 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.155519    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs" podUID="d2ecd98d-b049-4ce5-84f6-0546aed2d0f0" containerName="kube-rbac-proxy" containerID="cri-o://5f056fd53f0269d043d0801306e31a3ccc895b49981193b2f5278553bb5f5683" gracePeriod=30
Oct 30 03:40:41.228483 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.228455    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"]
Oct 30 03:40:41.228671 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.228644    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw" podUID="04fff530-1d0e-4e48-8d16-57bdba3a0ede" containerName="korrel8r" containerID="cri-o://978437718cea2ff216f3e59353d1b882c5fc64804d017e7e360cbfd294286f8b" gracePeriod=30
Oct 30 03:40:41.482324 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.482294    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:40:41.511254 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.511229    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"
Oct 30 03:40:41.656720 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.656681    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:41.661083 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.661059    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fff530-1d0e-4e48-8d16-57bdba3a0ede-kube-api-access-92vfb\" (UniqueName: \"kubernetes.io/projected/04fff530-1d0e-4e48-8d16-57bdba3a0ede-kube-api-access-92vfb\") pod \"04fff530-1d0e-4e48-8d16-57bdba3a0ede\" (UID: \"04fff530-1d0e-4e48-8d16-57bdba3a0ede\") "
Oct 30 03:40:41.661373 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.661101    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-client-cert\") pod \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") "
Oct 30 03:40:41.661373 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.661143    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-ca\") pod \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") "
Oct 30 03:40:41.661373 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.661235    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04fff530-1d0e-4e48-8d16-57bdba3a0ede-config\" (UniqueName: \"kubernetes.io/configmap/04fff530-1d0e-4e48-8d16-57bdba3a0ede-config\") pod \"04fff530-1d0e-4e48-8d16-57bdba3a0ede\" (UID: \"04fff530-1d0e-4e48-8d16-57bdba3a0ede\") "
Oct 30 03:40:41.661373 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.661280    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-tls\" (UniqueName: \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-tls\") pod \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") "
Oct 30 03:40:41.661373 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.661362    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-kube-api-access-bljmz\" (UniqueName: \"kubernetes.io/projected/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-kube-api-access-bljmz\") pod \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") "
Oct 30 03:40:41.662153 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.661833    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04fff530-1d0e-4e48-8d16-57bdba3a0ede-config" pod "04fff530-1d0e-4e48-8d16-57bdba3a0ede" (UID: "04fff530-1d0e-4e48-8d16-57bdba3a0ede"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:41.662153 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.661896    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-kube-rbac-proxy-config\") pod \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\" (UID: \"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0\") "
Oct 30 03:40:41.662153 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.661934    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04fff530-1d0e-4e48-8d16-57bdba3a0ede-tls\" (UniqueName: \"kubernetes.io/secret/04fff530-1d0e-4e48-8d16-57bdba3a0ede-tls\") pod \"04fff530-1d0e-4e48-8d16-57bdba3a0ede\" (UID: \"04fff530-1d0e-4e48-8d16-57bdba3a0ede\") "
Oct 30 03:40:41.671070 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.663625    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-ca" pod "d2ecd98d-b049-4ce5-84f6-0546aed2d0f0" (UID: "d2ecd98d-b049-4ce5-84f6-0546aed2d0f0"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:41.671070 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.664422    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fff530-1d0e-4e48-8d16-57bdba3a0ede-kube-api-access-92vfb" pod "04fff530-1d0e-4e48-8d16-57bdba3a0ede" (UID: "04fff530-1d0e-4e48-8d16-57bdba3a0ede"). InnerVolumeSpecName "kube-api-access-92vfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:41.671070 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.664628    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92vfb\" (UniqueName: \"kubernetes.io/projected/04fff530-1d0e-4e48-8d16-57bdba3a0ede-kube-api-access-92vfb\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:41.671070 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.664660    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-ca\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:41.671070 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.664675    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04fff530-1d0e-4e48-8d16-57bdba3a0ede-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:41.671070 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.666811    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-client-cert" pod "d2ecd98d-b049-4ce5-84f6-0546aed2d0f0" (UID: "d2ecd98d-b049-4ce5-84f6-0546aed2d0f0"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:41.671070 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.668442    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-tls" pod "d2ecd98d-b049-4ce5-84f6-0546aed2d0f0" (UID: "d2ecd98d-b049-4ce5-84f6-0546aed2d0f0"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:41.671070 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.668609    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04fff530-1d0e-4e48-8d16-57bdba3a0ede-tls" pod "04fff530-1d0e-4e48-8d16-57bdba3a0ede" (UID: "04fff530-1d0e-4e48-8d16-57bdba3a0ede"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:41.671070 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.668740    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-kube-api-access-bljmz" pod "d2ecd98d-b049-4ce5-84f6-0546aed2d0f0" (UID: "d2ecd98d-b049-4ce5-84f6-0546aed2d0f0"). InnerVolumeSpecName "kube-api-access-bljmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:41.671070 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.668829    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-kube-rbac-proxy-config" pod "d2ecd98d-b049-4ce5-84f6-0546aed2d0f0" (UID: "d2ecd98d-b049-4ce5-84f6-0546aed2d0f0"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:41.752652 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.752574    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:40:41.752889 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.752844    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="a07bc058-5869-4242-8c84-a9858cd7eae3" containerName="prometheus" containerID="cri-o://6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b" gracePeriod=600
Oct 30 03:40:41.753047 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.752915    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="a07bc058-5869-4242-8c84-a9858cd7eae3" containerName="thanos-sidecar" containerID="cri-o://50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50" gracePeriod=600
Oct 30 03:40:41.753047 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.752935    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="a07bc058-5869-4242-8c84-a9858cd7eae3" containerName="config-reloader" containerID="cri-o://a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8" gracePeriod=600
Oct 30 03:40:41.767590 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.767565    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:41.767691 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.767593    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bljmz\" (UniqueName: \"kubernetes.io/projected/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-kube-api-access-bljmz\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:41.767691 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.767604    4960 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-kube-rbac-proxy-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:41.767691 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.767616    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/04fff530-1d0e-4e48-8d16-57bdba3a0ede-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:41.767691 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.767628    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0-prometheus-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:41.768319 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.768297    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 03:40:41.768577 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.768553    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="d1b42492-cd77-4565-913b-94670d1be346" containerName="alertmanager" containerID="cri-o://fc49999a8678c4fbf80581cdbcadf453acc789bdeec2eb1f59e1c3967b067546" gracePeriod=120
Oct 30 03:40:41.768703 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.768640    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="d1b42492-cd77-4565-913b-94670d1be346" containerName="config-reloader" containerID="cri-o://6101bf1446e1b0bdf794364a18d68215f9f33693ecdb00f774acd5dcd02c4826" gracePeriod=120
Oct 30 03:40:41.811660 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.811636    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:41.939700 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:41.939667    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:42.033974 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.033951    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:40:42.097512 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.097478    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:42.154414 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.154385    4960 generic.go:356] "Generic (PLEG): container finished" podID="d2ecd98d-b049-4ce5-84f6-0546aed2d0f0" containerID="5f056fd53f0269d043d0801306e31a3ccc895b49981193b2f5278553bb5f5683" exitCode=0
Oct 30 03:40:42.154582 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.154432    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"
Oct 30 03:40:42.154582 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.154468    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs" event={"ID":"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0","Type":"ContainerDied","Data":"5f056fd53f0269d043d0801306e31a3ccc895b49981193b2f5278553bb5f5683"}
Oct 30 03:40:42.154582 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.154521    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs" event={"ID":"d2ecd98d-b049-4ce5-84f6-0546aed2d0f0","Type":"ContainerDied","Data":"9e125f95766066f484482743c0a9731f8693f165ff985639997df02ca6803993"}
Oct 30 03:40:42.154582 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.154552    4960 scope.go:122] "RemoveContainer" containerID="5f056fd53f0269d043d0801306e31a3ccc895b49981193b2f5278553bb5f5683"
Oct 30 03:40:42.155863 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.155840    4960 generic.go:356] "Generic (PLEG): container finished" podID="04fff530-1d0e-4e48-8d16-57bdba3a0ede" containerID="978437718cea2ff216f3e59353d1b882c5fc64804d017e7e360cbfd294286f8b" exitCode=0
Oct 30 03:40:42.156458 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.155881    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"
Oct 30 03:40:42.156458 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.155915    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw" event={"ID":"04fff530-1d0e-4e48-8d16-57bdba3a0ede","Type":"ContainerDied","Data":"978437718cea2ff216f3e59353d1b882c5fc64804d017e7e360cbfd294286f8b"}
Oct 30 03:40:42.156458 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.155964    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw" event={"ID":"04fff530-1d0e-4e48-8d16-57bdba3a0ede","Type":"ContainerDied","Data":"8736460cb10bbe65a7f9f2054ade62b6d7bac3234cb845ecf4f5a57836f62a61"}
Oct 30 03:40:42.159361 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.159339    4960 generic.go:356] "Generic (PLEG): container finished" podID="d1b42492-cd77-4565-913b-94670d1be346" containerID="6101bf1446e1b0bdf794364a18d68215f9f33693ecdb00f774acd5dcd02c4826" exitCode=0
Oct 30 03:40:42.159477 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.159411    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"d1b42492-cd77-4565-913b-94670d1be346","Type":"ContainerDied","Data":"6101bf1446e1b0bdf794364a18d68215f9f33693ecdb00f774acd5dcd02c4826"}
Oct 30 03:40:42.161895 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.161877    4960 generic.go:356] "Generic (PLEG): container finished" podID="a07bc058-5869-4242-8c84-a9858cd7eae3" containerID="50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50" exitCode=0
Oct 30 03:40:42.161895 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.161893    4960 generic.go:356] "Generic (PLEG): container finished" podID="a07bc058-5869-4242-8c84-a9858cd7eae3" containerID="a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8" exitCode=0
Oct 30 03:40:42.162028 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.161900    4960 generic.go:356] "Generic (PLEG): container finished" podID="a07bc058-5869-4242-8c84-a9858cd7eae3" containerID="6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b" exitCode=0
Oct 30 03:40:42.162028 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.161929    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a07bc058-5869-4242-8c84-a9858cd7eae3","Type":"ContainerDied","Data":"50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50"}
Oct 30 03:40:42.162028 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.161949    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:40:42.162028 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.161974    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a07bc058-5869-4242-8c84-a9858cd7eae3","Type":"ContainerDied","Data":"a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8"}
Oct 30 03:40:42.162028 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.162000    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a07bc058-5869-4242-8c84-a9858cd7eae3","Type":"ContainerDied","Data":"6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b"}
Oct 30 03:40:42.162028 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.162023    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a07bc058-5869-4242-8c84-a9858cd7eae3","Type":"ContainerDied","Data":"2d45a61e8ded1e7162027bbc9e3c60107f20dcffc18ad57d1eb5ebbef87cc42e"}
Oct 30 03:40:42.164879 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.164665    4960 scope.go:122] "RemoveContainer" containerID="5f056fd53f0269d043d0801306e31a3ccc895b49981193b2f5278553bb5f5683"
Oct 30 03:40:42.164954 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:40:42.164891    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f056fd53f0269d043d0801306e31a3ccc895b49981193b2f5278553bb5f5683\": container with ID starting with 5f056fd53f0269d043d0801306e31a3ccc895b49981193b2f5278553bb5f5683 not found: ID does not exist" containerID="5f056fd53f0269d043d0801306e31a3ccc895b49981193b2f5278553bb5f5683"
Oct 30 03:40:42.164954 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.164912    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f056fd53f0269d043d0801306e31a3ccc895b49981193b2f5278553bb5f5683"} err="failed to get container status \"5f056fd53f0269d043d0801306e31a3ccc895b49981193b2f5278553bb5f5683\": rpc error: code = NotFound desc = could not find container \"5f056fd53f0269d043d0801306e31a3ccc895b49981193b2f5278553bb5f5683\": container with ID starting with 5f056fd53f0269d043d0801306e31a3ccc895b49981193b2f5278553bb5f5683 not found: ID does not exist"
Oct 30 03:40:42.164954 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.164927    4960 scope.go:122] "RemoveContainer" containerID="978437718cea2ff216f3e59353d1b882c5fc64804d017e7e360cbfd294286f8b"
Oct 30 03:40:42.173711 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.173684    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-config\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-config\") pod \"a07bc058-5869-4242-8c84-a9858cd7eae3\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") "
Oct 30 03:40:42.173862 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.173761    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"a07bc058-5869-4242-8c84-a9858cd7eae3\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") "
Oct 30 03:40:42.173862 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.173791    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"a07bc058-5869-4242-8c84-a9858cd7eae3\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") "
Oct 30 03:40:42.173862 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.173841    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07bc058-5869-4242-8c84-a9858cd7eae3-kube-api-access-xs4h7\" (UniqueName: \"kubernetes.io/projected/a07bc058-5869-4242-8c84-a9858cd7eae3-kube-api-access-xs4h7\") pod \"a07bc058-5869-4242-8c84-a9858cd7eae3\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") "
Oct 30 03:40:42.174095 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.173880    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"a07bc058-5869-4242-8c84-a9858cd7eae3\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") "
Oct 30 03:40:42.174095 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.173907    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-thanos-prometheus-http-client-file\") pod \"a07bc058-5869-4242-8c84-a9858cd7eae3\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") "
Oct 30 03:40:42.174095 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.173938    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"a07bc058-5869-4242-8c84-a9858cd7eae3\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") "
Oct 30 03:40:42.174095 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.173979    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-secret-prometheus-web-tls-ca\") pod \"a07bc058-5869-4242-8c84-a9858cd7eae3\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") "
Oct 30 03:40:42.174095 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.174006    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"a07bc058-5869-4242-8c84-a9858cd7eae3\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") "
Oct 30 03:40:42.174095 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.174041    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07bc058-5869-4242-8c84-a9858cd7eae3-tls-assets\" (UniqueName: \"kubernetes.io/projected/a07bc058-5869-4242-8c84-a9858cd7eae3-tls-assets\") pod \"a07bc058-5869-4242-8c84-a9858cd7eae3\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") "
Oct 30 03:40:42.174377 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.174118    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config\") pod \"a07bc058-5869-4242-8c84-a9858cd7eae3\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") "
Oct 30 03:40:42.174377 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.174251    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"a07bc058-5869-4242-8c84-a9858cd7eae3\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") "
Oct 30 03:40:42.174377 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.174288    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07bc058-5869-4242-8c84-a9858cd7eae3-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a07bc058-5869-4242-8c84-a9858cd7eae3-config-out\") pod \"a07bc058-5869-4242-8c84-a9858cd7eae3\" (UID: \"a07bc058-5869-4242-8c84-a9858cd7eae3\") "
Oct 30 03:40:42.174858 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.174834    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-1" pod "a07bc058-5869-4242-8c84-a9858cd7eae3" (UID: "a07bc058-5869-4242-8c84-a9858cd7eae3"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:42.175389 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.175358    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-0" pod "a07bc058-5869-4242-8c84-a9858cd7eae3" (UID: "a07bc058-5869-4242-8c84-a9858cd7eae3"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:42.177390 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.177362    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07bc058-5869-4242-8c84-a9858cd7eae3-config-out" pod "a07bc058-5869-4242-8c84-a9858cd7eae3" (UID: "a07bc058-5869-4242-8c84-a9858cd7eae3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:42.178449 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.178422    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-secret-prometheus-web-tls-ca" pod "a07bc058-5869-4242-8c84-a9858cd7eae3" (UID: "a07bc058-5869-4242-8c84-a9858cd7eae3"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.179287 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.179259    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07bc058-5869-4242-8c84-a9858cd7eae3-tls-assets" pod "a07bc058-5869-4242-8c84-a9858cd7eae3" (UID: "a07bc058-5869-4242-8c84-a9858cd7eae3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:42.179386 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.179295    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "a07bc058-5869-4242-8c84-a9858cd7eae3" (UID: "a07bc058-5869-4242-8c84-a9858cd7eae3"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.179522 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.179500    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-thanos-prometheus-http-client-file" pod "a07bc058-5869-4242-8c84-a9858cd7eae3" (UID: "a07bc058-5869-4242-8c84-a9858cd7eae3"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.179719 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.179566    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-2" pod "a07bc058-5869-4242-8c84-a9858cd7eae3" (UID: "a07bc058-5869-4242-8c84-a9858cd7eae3"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:42.179914 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.179881    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-config" pod "a07bc058-5869-4242-8c84-a9858cd7eae3" (UID: "a07bc058-5869-4242-8c84-a9858cd7eae3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.180978 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.180953    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "a07bc058-5869-4242-8c84-a9858cd7eae3" (UID: "a07bc058-5869-4242-8c84-a9858cd7eae3"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.181360 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.181340    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07bc058-5869-4242-8c84-a9858cd7eae3-kube-api-access-xs4h7" pod "a07bc058-5869-4242-8c84-a9858cd7eae3" (UID: "a07bc058-5869-4242-8c84-a9858cd7eae3"). InnerVolumeSpecName "kube-api-access-xs4h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:42.183459 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.183436    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87" pod "a07bc058-5869-4242-8c84-a9858cd7eae3" (UID: "a07bc058-5869-4242-8c84-a9858cd7eae3"). InnerVolumeSpecName "pvc-9689c530-7f91-4584-b28e-8809bed9e799". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:40:42.191373 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.191353    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config" pod "a07bc058-5869-4242-8c84-a9858cd7eae3" (UID: "a07bc058-5869-4242-8c84-a9858cd7eae3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.206778 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.206746    4960 scope.go:122] "RemoveContainer" containerID="978437718cea2ff216f3e59353d1b882c5fc64804d017e7e360cbfd294286f8b"
Oct 30 03:40:42.207118 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:40:42.207077    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"978437718cea2ff216f3e59353d1b882c5fc64804d017e7e360cbfd294286f8b\": container with ID starting with 978437718cea2ff216f3e59353d1b882c5fc64804d017e7e360cbfd294286f8b not found: ID does not exist" containerID="978437718cea2ff216f3e59353d1b882c5fc64804d017e7e360cbfd294286f8b"
Oct 30 03:40:42.207118 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.207105    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"978437718cea2ff216f3e59353d1b882c5fc64804d017e7e360cbfd294286f8b"} err="failed to get container status \"978437718cea2ff216f3e59353d1b882c5fc64804d017e7e360cbfd294286f8b\": rpc error: code = NotFound desc = could not find container \"978437718cea2ff216f3e59353d1b882c5fc64804d017e7e360cbfd294286f8b\": container with ID starting with 978437718cea2ff216f3e59353d1b882c5fc64804d017e7e360cbfd294286f8b not found: ID does not exist"
Oct 30 03:40:42.207278 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.207121    4960 scope.go:122] "RemoveContainer" containerID="50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50"
Oct 30 03:40:42.207626 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.207605    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:42.215779 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.215753    4960 scope.go:122] "RemoveContainer" containerID="a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8"
Oct 30 03:40:42.223684 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.223663    4960 scope.go:122] "RemoveContainer" containerID="6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b"
Oct 30 03:40:42.231527 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.231509    4960 scope.go:122] "RemoveContainer" containerID="0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba"
Oct 30 03:40:42.239190 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.239159    4960 scope.go:122] "RemoveContainer" containerID="50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50"
Oct 30 03:40:42.239447 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:40:42.239431    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50\": container with ID starting with 50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50 not found: ID does not exist" containerID="50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50"
Oct 30 03:40:42.239503 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.239454    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50"} err="failed to get container status \"50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50\": rpc error: code = NotFound desc = could not find container \"50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50\": container with ID starting with 50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50 not found: ID does not exist"
Oct 30 03:40:42.239503 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.239468    4960 scope.go:122] "RemoveContainer" containerID="a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8"
Oct 30 03:40:42.239685 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:40:42.239670    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8\": container with ID starting with a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8 not found: ID does not exist" containerID="a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8"
Oct 30 03:40:42.239729 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.239689    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8"} err="failed to get container status \"a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8\": rpc error: code = NotFound desc = could not find container \"a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8\": container with ID starting with a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8 not found: ID does not exist"
Oct 30 03:40:42.239729 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.239701    4960 scope.go:122] "RemoveContainer" containerID="6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b"
Oct 30 03:40:42.239897 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:40:42.239882    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b\": container with ID starting with 6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b not found: ID does not exist" containerID="6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b"
Oct 30 03:40:42.239940 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.239900    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b"} err="failed to get container status \"6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b\": rpc error: code = NotFound desc = could not find container \"6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b\": container with ID starting with 6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b not found: ID does not exist"
Oct 30 03:40:42.239940 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.239912    4960 scope.go:122] "RemoveContainer" containerID="0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba"
Oct 30 03:40:42.240135 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:40:42.240120    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba\": container with ID starting with 0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba not found: ID does not exist" containerID="0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba"
Oct 30 03:40:42.240193 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.240138    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba"} err="failed to get container status \"0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba\": rpc error: code = NotFound desc = could not find container \"0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba\": container with ID starting with 0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba not found: ID does not exist"
Oct 30 03:40:42.240193 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.240149    4960 scope.go:122] "RemoveContainer" containerID="50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50"
Oct 30 03:40:42.240404 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.240383    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50"} err="failed to get container status \"50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50\": rpc error: code = NotFound desc = could not find container \"50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50\": container with ID starting with 50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50 not found: ID does not exist"
Oct 30 03:40:42.240452 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.240405    4960 scope.go:122] "RemoveContainer" containerID="a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8"
Oct 30 03:40:42.240605 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.240590    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8"} err="failed to get container status \"a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8\": rpc error: code = NotFound desc = could not find container \"a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8\": container with ID starting with a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8 not found: ID does not exist"
Oct 30 03:40:42.240652 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.240605    4960 scope.go:122] "RemoveContainer" containerID="6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b"
Oct 30 03:40:42.240794 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.240779    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b"} err="failed to get container status \"6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b\": rpc error: code = NotFound desc = could not find container \"6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b\": container with ID starting with 6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b not found: ID does not exist"
Oct 30 03:40:42.240841 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.240794    4960 scope.go:122] "RemoveContainer" containerID="0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba"
Oct 30 03:40:42.240973 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.240958    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba"} err="failed to get container status \"0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba\": rpc error: code = NotFound desc = could not find container \"0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba\": container with ID starting with 0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba not found: ID does not exist"
Oct 30 03:40:42.241023 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.240973    4960 scope.go:122] "RemoveContainer" containerID="50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50"
Oct 30 03:40:42.241214 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.241199    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50"} err="failed to get container status \"50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50\": rpc error: code = NotFound desc = could not find container \"50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50\": container with ID starting with 50a8b55aa985f195803ca57cc57d3269c9449273d041cd3b2d3aa8aadce00a50 not found: ID does not exist"
Oct 30 03:40:42.241253 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.241214    4960 scope.go:122] "RemoveContainer" containerID="a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8"
Oct 30 03:40:42.241421 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.241405    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8"} err="failed to get container status \"a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8\": rpc error: code = NotFound desc = could not find container \"a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8\": container with ID starting with a1760c35418abadacf6bc6d910531fdd268483b1fe5822759f6c8e62b35a81e8 not found: ID does not exist"
Oct 30 03:40:42.241471 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.241421    4960 scope.go:122] "RemoveContainer" containerID="6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b"
Oct 30 03:40:42.241605 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.241589    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b"} err="failed to get container status \"6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b\": rpc error: code = NotFound desc = could not find container \"6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b\": container with ID starting with 6ab9493e998f82c087b820048cfea7d2a1c695eeebc957fe64f1fc72a19a7a8b not found: ID does not exist"
Oct 30 03:40:42.241605 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.241603    4960 scope.go:122] "RemoveContainer" containerID="0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba"
Oct 30 03:40:42.241780 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.241765    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba"} err="failed to get container status \"0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba\": rpc error: code = NotFound desc = could not find container \"0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba\": container with ID starting with 0a36f29317d52a0758ea1a734ccb2ecd385a873979dec342110a517b8c918bba not found: ID does not exist"
Oct 30 03:40:42.245652 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.245632    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"]
Oct 30 03:40:42.249332 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.249314    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-wb2rw"]
Oct 30 03:40:42.275514 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.275462    4960 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") on node \"ip-10-0-128-241.ec2.internal\" "
Oct 30 03:40:42.275514 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.275484    4960 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a07bc058-5869-4242-8c84-a9858cd7eae3-config-out\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.275514 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.275494    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.275514 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.275503    4960 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.275514 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.275514    4960 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.275750 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.275524    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs4h7\" (UniqueName: \"kubernetes.io/projected/a07bc058-5869-4242-8c84-a9858cd7eae3-kube-api-access-xs4h7\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.275750 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.275538    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.275750 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.275549    4960 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.275750 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.275558    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.275750 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.275566    4960 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-secret-prometheus-web-tls-ca\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.275750 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.275575    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a07bc058-5869-4242-8c84-a9858cd7eae3-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.275750 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.275584    4960 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a07bc058-5869-4242-8c84-a9858cd7eae3-tls-assets\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.275750 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.275593    4960 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a07bc058-5869-4242-8c84-a9858cd7eae3-web-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.285723 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.285704    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"]
Oct 30 03:40:42.287497 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.287483    4960 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-9689c530-7f91-4584-b28e-8809bed9e799" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87") on node "ip-10-0-128-241.ec2.internal"
Oct 30 03:40:42.291890 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.291865    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-kfvxs"]
Oct 30 03:40:42.376114 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.376092    4960 reconciler_common.go:299] "Volume detached for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"csi-6f8c8e4c1326657251c30594036033694cadf25e97408e4f9d9da436f78f3fa4\""
Oct 30 03:40:42.471592 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.471558    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:40:42.472114 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.472080    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="ce464ccc-6523-4de8-9d11-892149d242cd" containerName="otc-container" containerID="cri-o://07e375d347c3e033ef9ce5b714fa9f4198d5cac840d478c9f062e9d5640fdd26" gracePeriod=30
Oct 30 03:40:42.546539 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.546385    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:42.568684 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.568658    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:40:42.572635 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.572609    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:40:42.678693 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.678655    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:42.708367 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.708344    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:40:42.879312 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.879279    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce464ccc-6523-4de8-9d11-892149d242cd-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ce464ccc-6523-4de8-9d11-892149d242cd-otc-internal\") pod \"ce464ccc-6523-4de8-9d11-892149d242cd\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") "
Oct 30 03:40:42.879479 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.879379    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce464ccc-6523-4de8-9d11-892149d242cd-kube-api-access-lj4l4\" (UniqueName: \"kubernetes.io/projected/ce464ccc-6523-4de8-9d11-892149d242cd-kube-api-access-lj4l4\") pod \"ce464ccc-6523-4de8-9d11-892149d242cd\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") "
Oct 30 03:40:42.879479 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.879411    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-otlp-tls-certs\") pod \"ce464ccc-6523-4de8-9d11-892149d242cd\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") "
Oct 30 03:40:42.879479 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.879456    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-data-science-collector-ta-client-cert\") pod \"ce464ccc-6523-4de8-9d11-892149d242cd\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") "
Oct 30 03:40:42.879663 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.879569    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-tls-certs\") pod \"ce464ccc-6523-4de8-9d11-892149d242cd\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") "
Oct 30 03:40:42.879663 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.879620    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-monitoring-tls-certs\") pod \"ce464ccc-6523-4de8-9d11-892149d242cd\" (UID: \"ce464ccc-6523-4de8-9d11-892149d242cd\") "
Oct 30 03:40:42.879769 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.879711    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce464ccc-6523-4de8-9d11-892149d242cd-otc-internal" pod "ce464ccc-6523-4de8-9d11-892149d242cd" (UID: "ce464ccc-6523-4de8-9d11-892149d242cd"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:42.879933 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.879914    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ce464ccc-6523-4de8-9d11-892149d242cd-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.881577 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.881556    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce464ccc-6523-4de8-9d11-892149d242cd-kube-api-access-lj4l4" pod "ce464ccc-6523-4de8-9d11-892149d242cd" (UID: "ce464ccc-6523-4de8-9d11-892149d242cd"). InnerVolumeSpecName "kube-api-access-lj4l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:42.882164 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.882143    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-otlp-tls-certs" pod "ce464ccc-6523-4de8-9d11-892149d242cd" (UID: "ce464ccc-6523-4de8-9d11-892149d242cd"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.882164 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.882155    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-data-science-collector-ta-client-cert" pod "ce464ccc-6523-4de8-9d11-892149d242cd" (UID: "ce464ccc-6523-4de8-9d11-892149d242cd"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.882367 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.882245    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-tls-certs" pod "ce464ccc-6523-4de8-9d11-892149d242cd" (UID: "ce464ccc-6523-4de8-9d11-892149d242cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.882367 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.882287    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-monitoring-tls-certs" pod "ce464ccc-6523-4de8-9d11-892149d242cd" (UID: "ce464ccc-6523-4de8-9d11-892149d242cd"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.981037 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.981011    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj4l4\" (UniqueName: \"kubernetes.io/projected/ce464ccc-6523-4de8-9d11-892149d242cd-kube-api-access-lj4l4\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.981037 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.981034    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.981293 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.981046    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.981293 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.981055    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.981293 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:42.981063    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce464ccc-6523-4de8-9d11-892149d242cd-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:43.171453 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:43.171420    4960 generic.go:356] "Generic (PLEG): container finished" podID="ce464ccc-6523-4de8-9d11-892149d242cd" containerID="07e375d347c3e033ef9ce5b714fa9f4198d5cac840d478c9f062e9d5640fdd26" exitCode=0
Oct 30 03:40:43.171814 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:43.171475    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:40:43.171814 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:43.171499    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ce464ccc-6523-4de8-9d11-892149d242cd","Type":"ContainerDied","Data":"07e375d347c3e033ef9ce5b714fa9f4198d5cac840d478c9f062e9d5640fdd26"}
Oct 30 03:40:43.171814 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:43.171563    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ce464ccc-6523-4de8-9d11-892149d242cd","Type":"ContainerDied","Data":"d35114c87a2f167018fd69225bc048e9d86c94eb9045f5dad29aa3bdbca64bca"}
Oct 30 03:40:43.171814 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:43.171594    4960 scope.go:122] "RemoveContainer" containerID="07e375d347c3e033ef9ce5b714fa9f4198d5cac840d478c9f062e9d5640fdd26"
Oct 30 03:40:43.180878 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:43.180863    4960 scope.go:122] "RemoveContainer" containerID="07e375d347c3e033ef9ce5b714fa9f4198d5cac840d478c9f062e9d5640fdd26"
Oct 30 03:40:43.181124 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:40:43.181106    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e375d347c3e033ef9ce5b714fa9f4198d5cac840d478c9f062e9d5640fdd26\": container with ID starting with 07e375d347c3e033ef9ce5b714fa9f4198d5cac840d478c9f062e9d5640fdd26 not found: ID does not exist" containerID="07e375d347c3e033ef9ce5b714fa9f4198d5cac840d478c9f062e9d5640fdd26"
Oct 30 03:40:43.181171 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:43.181133    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e375d347c3e033ef9ce5b714fa9f4198d5cac840d478c9f062e9d5640fdd26"} err="failed to get container status \"07e375d347c3e033ef9ce5b714fa9f4198d5cac840d478c9f062e9d5640fdd26\": rpc error: code = NotFound desc = could not find container \"07e375d347c3e033ef9ce5b714fa9f4198d5cac840d478c9f062e9d5640fdd26\": container with ID starting with 07e375d347c3e033ef9ce5b714fa9f4198d5cac840d478c9f062e9d5640fdd26 not found: ID does not exist"
Oct 30 03:40:43.225685 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:43.225660    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:40:43.229250 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:43.229221    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:40:43.409782 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:43.409753    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="d1b42492-cd77-4565-913b-94670d1be346" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.37:9093/-/ready\": dial tcp 10.132.0.37:9093: connect: connection refused"
Oct 30 03:40:43.503768 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:43.503705    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04fff530-1d0e-4e48-8d16-57bdba3a0ede" path="/var/lib/kubelet/pods/04fff530-1d0e-4e48-8d16-57bdba3a0ede/volumes"
Oct 30 03:40:43.504210 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:43.504193    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07bc058-5869-4242-8c84-a9858cd7eae3" path="/var/lib/kubelet/pods/a07bc058-5869-4242-8c84-a9858cd7eae3/volumes"
Oct 30 03:40:43.505006 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:43.504991    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce464ccc-6523-4de8-9d11-892149d242cd" path="/var/lib/kubelet/pods/ce464ccc-6523-4de8-9d11-892149d242cd/volumes"
Oct 30 03:40:43.505484 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:43.505469    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2ecd98d-b049-4ce5-84f6-0546aed2d0f0" path="/var/lib/kubelet/pods/d2ecd98d-b049-4ce5-84f6-0546aed2d0f0/volumes"
Oct 30 03:40:43.544663 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:43.544615    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:43.661192 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:43.661132    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:43.775721 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:43.775611    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:44.204011 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:44.203953    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:44.323342 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:44.323290    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:47.625472 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.625433    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:47.673533 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.673503    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-mxg24"]
Oct 30 03:40:47.673839 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.673827    4960 cpu_manager.go:414] "containerMap: removing container" podUID="d2ecd98d-b049-4ce5-84f6-0546aed2d0f0" containerName="kube-rbac-proxy"
Oct 30 03:40:47.673882 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.673841    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ecd98d-b049-4ce5-84f6-0546aed2d0f0" containerName="kube-rbac-proxy"
Oct 30 03:40:47.673882 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.673849    4960 cpu_manager.go:414] "containerMap: removing container" podUID="ce464ccc-6523-4de8-9d11-892149d242cd" containerName="otc-container"
Oct 30 03:40:47.673882 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.673855    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce464ccc-6523-4de8-9d11-892149d242cd" containerName="otc-container"
Oct 30 03:40:47.673882 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.673864    4960 cpu_manager.go:414] "containerMap: removing container" podUID="a07bc058-5869-4242-8c84-a9858cd7eae3" containerName="thanos-sidecar"
Oct 30 03:40:47.673882 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.673869    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07bc058-5869-4242-8c84-a9858cd7eae3" containerName="thanos-sidecar"
Oct 30 03:40:47.673882 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.673882    4960 cpu_manager.go:414] "containerMap: removing container" podUID="04fff530-1d0e-4e48-8d16-57bdba3a0ede" containerName="korrel8r"
Oct 30 03:40:47.674059 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.673887    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04fff530-1d0e-4e48-8d16-57bdba3a0ede" containerName="korrel8r"
Oct 30 03:40:47.674059 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.673896    4960 cpu_manager.go:414] "containerMap: removing container" podUID="a07bc058-5869-4242-8c84-a9858cd7eae3" containerName="prometheus"
Oct 30 03:40:47.674059 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.673902    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07bc058-5869-4242-8c84-a9858cd7eae3" containerName="prometheus"
Oct 30 03:40:47.674059 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.673917    4960 cpu_manager.go:414] "containerMap: removing container" podUID="a07bc058-5869-4242-8c84-a9858cd7eae3" containerName="init-config-reloader"
Oct 30 03:40:47.674059 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.673921    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07bc058-5869-4242-8c84-a9858cd7eae3" containerName="init-config-reloader"
Oct 30 03:40:47.674059 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.673928    4960 cpu_manager.go:414] "containerMap: removing container" podUID="a07bc058-5869-4242-8c84-a9858cd7eae3" containerName="config-reloader"
Oct 30 03:40:47.674059 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.673932    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07bc058-5869-4242-8c84-a9858cd7eae3" containerName="config-reloader"
Oct 30 03:40:47.674059 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.673982    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07bc058-5869-4242-8c84-a9858cd7eae3" containerName="config-reloader"
Oct 30 03:40:47.674059 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.673991    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ecd98d-b049-4ce5-84f6-0546aed2d0f0" containerName="kube-rbac-proxy"
Oct 30 03:40:47.674059 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.673996    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce464ccc-6523-4de8-9d11-892149d242cd" containerName="otc-container"
Oct 30 03:40:47.674059 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.674003    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07bc058-5869-4242-8c84-a9858cd7eae3" containerName="thanos-sidecar"
Oct 30 03:40:47.674059 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.674009    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="04fff530-1d0e-4e48-8d16-57bdba3a0ede" containerName="korrel8r"
Oct 30 03:40:47.674059 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.674015    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07bc058-5869-4242-8c84-a9858cd7eae3" containerName="prometheus"
Oct 30 03:40:47.678511 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.678494    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24"
Oct 30 03:40:47.695910 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.695886    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-mxg24"]
Oct 30 03:40:47.708502 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.708482    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\""
Oct 30 03:40:47.708502 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.708496    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\""
Oct 30 03:40:47.708647 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.708489    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-bmnbg\""
Oct 30 03:40:47.735328 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.735305    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:47.814704 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.814684    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/595feb8a-8943-4722-a57c-40b9a4217941-tls\") pod \"korrel8r-6644769df5-mxg24\" (UID: \"595feb8a-8943-4722-a57c-40b9a4217941\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24"
Oct 30 03:40:47.814817 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.814725    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/595feb8a-8943-4722-a57c-40b9a4217941-config\") pod \"korrel8r-6644769df5-mxg24\" (UID: \"595feb8a-8943-4722-a57c-40b9a4217941\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24"
Oct 30 03:40:47.814817 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.814746    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8lv6\" (UniqueName: \"kubernetes.io/projected/595feb8a-8943-4722-a57c-40b9a4217941-kube-api-access-r8lv6\") pod \"korrel8r-6644769df5-mxg24\" (UID: \"595feb8a-8943-4722-a57c-40b9a4217941\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24"
Oct 30 03:40:47.915936 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.915886    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/595feb8a-8943-4722-a57c-40b9a4217941-tls\") pod \"korrel8r-6644769df5-mxg24\" (UID: \"595feb8a-8943-4722-a57c-40b9a4217941\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24"
Oct 30 03:40:47.915936 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.915926    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/595feb8a-8943-4722-a57c-40b9a4217941-config\") pod \"korrel8r-6644769df5-mxg24\" (UID: \"595feb8a-8943-4722-a57c-40b9a4217941\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24"
Oct 30 03:40:47.916092 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.915945    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8lv6\" (UniqueName: \"kubernetes.io/projected/595feb8a-8943-4722-a57c-40b9a4217941-kube-api-access-r8lv6\") pod \"korrel8r-6644769df5-mxg24\" (UID: \"595feb8a-8943-4722-a57c-40b9a4217941\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24"
Oct 30 03:40:47.916651 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.916627    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/595feb8a-8943-4722-a57c-40b9a4217941-config\") pod \"korrel8r-6644769df5-mxg24\" (UID: \"595feb8a-8943-4722-a57c-40b9a4217941\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24"
Oct 30 03:40:47.918421 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.918402    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/595feb8a-8943-4722-a57c-40b9a4217941-tls\") pod \"korrel8r-6644769df5-mxg24\" (UID: \"595feb8a-8943-4722-a57c-40b9a4217941\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24"
Oct 30 03:40:47.936728 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.936704    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8lv6\" (UniqueName: \"kubernetes.io/projected/595feb8a-8943-4722-a57c-40b9a4217941-kube-api-access-r8lv6\") pod \"korrel8r-6644769df5-mxg24\" (UID: \"595feb8a-8943-4722-a57c-40b9a4217941\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24"
Oct 30 03:40:47.990613 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:47.990586    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24"
Oct 30 03:40:48.280694 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.280668    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-mxg24"]
Oct 30 03:40:48.282457 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:40:48.282428    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595feb8a_8943_4722_a57c_40b9a4217941.slice/crio-5db71151de6846793ebea20d8971a52efbf4c509733145f0fb486fc367c77e6f WatchSource:0}: Error finding container 5db71151de6846793ebea20d8971a52efbf4c509733145f0fb486fc367c77e6f: Status 404 returned error can't find the container with id 5db71151de6846793ebea20d8971a52efbf4c509733145f0fb486fc367c77e6f
Oct 30 03:40:48.398139 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.398108    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:40:48.401474 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.401450    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:40:48.409156 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.408994    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="d1b42492-cd77-4565-913b-94670d1be346" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.37:9093/-/ready\": dial tcp 10.132.0.37:9093: connect: connection refused"
Oct 30 03:40:48.410869 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.410842    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:48.420784 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.420764    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:40:48.465130 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.465093    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:40:48.465436 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.465417    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-4tr57\""
Oct 30 03:40:48.465537 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.465515    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:40:48.487281 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.487263    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\""
Oct 30 03:40:48.520662 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.520613    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:40:48.520770 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.520666    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrkmr\" (UniqueName: \"kubernetes.io/projected/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-kube-api-access-vrkmr\") pod \"data-science-collector-collector-1\" (UID: \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:40:48.520822 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.520767    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:40:48.520880 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.520842    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:40:48.524715 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.524694    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:48.622081 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.622057    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:40:48.622217 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.622093    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:40:48.622217 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.622124    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrkmr\" (UniqueName: \"kubernetes.io/projected/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-kube-api-access-vrkmr\") pod \"data-science-collector-collector-1\" (UID: \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:40:48.622217 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.622147    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:40:48.622748 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.622725    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:40:48.624852 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.624832    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:40:48.624931 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.624855    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:40:48.642442 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.642417    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrkmr\" (UniqueName: \"kubernetes.io/projected/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-kube-api-access-vrkmr\") pod \"data-science-collector-collector-1\" (UID: \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:40:48.654234 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.654207    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:48.717878 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:48.717855    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:40:49.015376 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:49.015354    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:40:49.017629 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:40:49.017594    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb86fdf7_abf2_4ab5_aa14_6a5e98563455.slice/crio-5998330fd70f5e0c07fe25f8c93b05dadf3e86bf4e33791aeadea6171eb9816b WatchSource:0}: Error finding container 5998330fd70f5e0c07fe25f8c93b05dadf3e86bf4e33791aeadea6171eb9816b: Status 404 returned error can't find the container with id 5998330fd70f5e0c07fe25f8c93b05dadf3e86bf4e33791aeadea6171eb9816b
Oct 30 03:40:49.020332 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:49.020298    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:40:49.020412 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:49.020361    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:40:49.201643 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:49.201605    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fb86fdf7-abf2-4ab5-aa14-6a5e98563455","Type":"ContainerStarted","Data":"315072835b14a8da0095223bb606919b2f2ad10aafc28228ec044a4ec930ed68"}
Oct 30 03:40:49.201832 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:49.201662    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fb86fdf7-abf2-4ab5-aa14-6a5e98563455","Type":"ContainerStarted","Data":"5998330fd70f5e0c07fe25f8c93b05dadf3e86bf4e33791aeadea6171eb9816b"}
Oct 30 03:40:49.203147 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:49.203122    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24" event={"ID":"595feb8a-8943-4722-a57c-40b9a4217941","Type":"ContainerStarted","Data":"009c07115f3fc685743a592236bb8b6abea953eb694ea02e25324f9471b9e868"}
Oct 30 03:40:49.203262 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:49.203164    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24" event={"ID":"595feb8a-8943-4722-a57c-40b9a4217941","Type":"ContainerStarted","Data":"5db71151de6846793ebea20d8971a52efbf4c509733145f0fb486fc367c77e6f"}
Oct 30 03:40:49.203315 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:49.203268    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24"
Oct 30 03:40:49.246557 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:49.246517    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.246503509 podStartE2EDuration="1.246503509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:49.24543213 +0000 UTC m=+3529.894480585" watchObservedRunningTime="2026-10-30 03:40:49.246503509 +0000 UTC m=+3529.895551990"
Oct 30 03:40:49.288885 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:49.288782    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24" podStartSLOduration=2.288768332 podStartE2EDuration="2.288768332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:49.287921008 +0000 UTC m=+3529.936969429" watchObservedRunningTime="2026-10-30 03:40:49.288768332 +0000 UTC m=+3529.937816752"
Oct 30 03:40:49.343122 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:49.343088    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:49.468440 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:49.468398    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:52.033586 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.033535    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:40:52.149732 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.149703    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b42492-cd77-4565-913b-94670d1be346-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/d1b42492-cd77-4565-913b-94670d1be346-alertmanager-data-science-monitoringstack-db\") pod \"d1b42492-cd77-4565-913b-94670d1be346\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") "
Oct 30 03:40:52.149882 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.149747    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-web-config\" (UniqueName: \"kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-web-config\") pod \"d1b42492-cd77-4565-913b-94670d1be346\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") "
Oct 30 03:40:52.149882 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.149778    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-cluster-tls-config\") pod \"d1b42492-cd77-4565-913b-94670d1be346\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") "
Oct 30 03:40:52.149882 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.149814    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b42492-cd77-4565-913b-94670d1be346-tls-assets\" (UniqueName: \"kubernetes.io/projected/d1b42492-cd77-4565-913b-94670d1be346-tls-assets\") pod \"d1b42492-cd77-4565-913b-94670d1be346\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") "
Oct 30 03:40:52.149882 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.149831    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b42492-cd77-4565-913b-94670d1be346-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1b42492-cd77-4565-913b-94670d1be346-config-out\") pod \"d1b42492-cd77-4565-913b-94670d1be346\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") "
Oct 30 03:40:52.149882 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.149873    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b42492-cd77-4565-913b-94670d1be346-kube-api-access-lzx5h\" (UniqueName: \"kubernetes.io/projected/d1b42492-cd77-4565-913b-94670d1be346-kube-api-access-lzx5h\") pod \"d1b42492-cd77-4565-913b-94670d1be346\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") "
Oct 30 03:40:52.150140 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.149954    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-config-volume\" (UniqueName: \"kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-config-volume\") pod \"d1b42492-cd77-4565-913b-94670d1be346\" (UID: \"d1b42492-cd77-4565-913b-94670d1be346\") "
Oct 30 03:40:52.150140 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.150034    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b42492-cd77-4565-913b-94670d1be346-alertmanager-data-science-monitoringstack-db" pod "d1b42492-cd77-4565-913b-94670d1be346" (UID: "d1b42492-cd77-4565-913b-94670d1be346"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:52.150288 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.150251    4960 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/d1b42492-cd77-4565-913b-94670d1be346-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:52.152340 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.152312    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b42492-cd77-4565-913b-94670d1be346-config-out" pod "d1b42492-cd77-4565-913b-94670d1be346" (UID: "d1b42492-cd77-4565-913b-94670d1be346"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:52.152640 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.152619    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b42492-cd77-4565-913b-94670d1be346-kube-api-access-lzx5h" pod "d1b42492-cd77-4565-913b-94670d1be346" (UID: "d1b42492-cd77-4565-913b-94670d1be346"). InnerVolumeSpecName "kube-api-access-lzx5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:52.152752 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.152717    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-config-volume" pod "d1b42492-cd77-4565-913b-94670d1be346" (UID: "d1b42492-cd77-4565-913b-94670d1be346"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:52.152836 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.152815    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b42492-cd77-4565-913b-94670d1be346-tls-assets" pod "d1b42492-cd77-4565-913b-94670d1be346" (UID: "d1b42492-cd77-4565-913b-94670d1be346"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:52.157772 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.157743    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-cluster-tls-config" pod "d1b42492-cd77-4565-913b-94670d1be346" (UID: "d1b42492-cd77-4565-913b-94670d1be346"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:52.165101 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.165075    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-web-config" pod "d1b42492-cd77-4565-913b-94670d1be346" (UID: "d1b42492-cd77-4565-913b-94670d1be346"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:52.220703 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.220674    4960 generic.go:356] "Generic (PLEG): container finished" podID="d1b42492-cd77-4565-913b-94670d1be346" containerID="fc49999a8678c4fbf80581cdbcadf453acc789bdeec2eb1f59e1c3967b067546" exitCode=0
Oct 30 03:40:52.220819 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.220745    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"d1b42492-cd77-4565-913b-94670d1be346","Type":"ContainerDied","Data":"fc49999a8678c4fbf80581cdbcadf453acc789bdeec2eb1f59e1c3967b067546"}
Oct 30 03:40:52.220819 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.220747    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:40:52.220819 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.220815    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"d1b42492-cd77-4565-913b-94670d1be346","Type":"ContainerDied","Data":"a2483d70b840c0f9470fb3908bca2aa9d2b62ef8c4b2b1f99ed737fede9d4744"}
Oct 30 03:40:52.220984 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.220840    4960 scope.go:122] "RemoveContainer" containerID="6101bf1446e1b0bdf794364a18d68215f9f33693ecdb00f774acd5dcd02c4826"
Oct 30 03:40:52.229795 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.229781    4960 scope.go:122] "RemoveContainer" containerID="fc49999a8678c4fbf80581cdbcadf453acc789bdeec2eb1f59e1c3967b067546"
Oct 30 03:40:52.237675 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.237649    4960 scope.go:122] "RemoveContainer" containerID="bc627a37430812e902275cf518392edaadf57747b423929d22628464856db1dd"
Oct 30 03:40:52.245307 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.245286    4960 scope.go:122] "RemoveContainer" containerID="6101bf1446e1b0bdf794364a18d68215f9f33693ecdb00f774acd5dcd02c4826"
Oct 30 03:40:52.245560 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:40:52.245539    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6101bf1446e1b0bdf794364a18d68215f9f33693ecdb00f774acd5dcd02c4826\": container with ID starting with 6101bf1446e1b0bdf794364a18d68215f9f33693ecdb00f774acd5dcd02c4826 not found: ID does not exist" containerID="6101bf1446e1b0bdf794364a18d68215f9f33693ecdb00f774acd5dcd02c4826"
Oct 30 03:40:52.245632 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.245571    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6101bf1446e1b0bdf794364a18d68215f9f33693ecdb00f774acd5dcd02c4826"} err="failed to get container status \"6101bf1446e1b0bdf794364a18d68215f9f33693ecdb00f774acd5dcd02c4826\": rpc error: code = NotFound desc = could not find container \"6101bf1446e1b0bdf794364a18d68215f9f33693ecdb00f774acd5dcd02c4826\": container with ID starting with 6101bf1446e1b0bdf794364a18d68215f9f33693ecdb00f774acd5dcd02c4826 not found: ID does not exist"
Oct 30 03:40:52.245632 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.245590    4960 scope.go:122] "RemoveContainer" containerID="fc49999a8678c4fbf80581cdbcadf453acc789bdeec2eb1f59e1c3967b067546"
Oct 30 03:40:52.245851 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:40:52.245831    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc49999a8678c4fbf80581cdbcadf453acc789bdeec2eb1f59e1c3967b067546\": container with ID starting with fc49999a8678c4fbf80581cdbcadf453acc789bdeec2eb1f59e1c3967b067546 not found: ID does not exist" containerID="fc49999a8678c4fbf80581cdbcadf453acc789bdeec2eb1f59e1c3967b067546"
Oct 30 03:40:52.245917 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.245857    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc49999a8678c4fbf80581cdbcadf453acc789bdeec2eb1f59e1c3967b067546"} err="failed to get container status \"fc49999a8678c4fbf80581cdbcadf453acc789bdeec2eb1f59e1c3967b067546\": rpc error: code = NotFound desc = could not find container \"fc49999a8678c4fbf80581cdbcadf453acc789bdeec2eb1f59e1c3967b067546\": container with ID starting with fc49999a8678c4fbf80581cdbcadf453acc789bdeec2eb1f59e1c3967b067546 not found: ID does not exist"
Oct 30 03:40:52.245917 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.245873    4960 scope.go:122] "RemoveContainer" containerID="bc627a37430812e902275cf518392edaadf57747b423929d22628464856db1dd"
Oct 30 03:40:52.246127 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:40:52.246113    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc627a37430812e902275cf518392edaadf57747b423929d22628464856db1dd\": container with ID starting with bc627a37430812e902275cf518392edaadf57747b423929d22628464856db1dd not found: ID does not exist" containerID="bc627a37430812e902275cf518392edaadf57747b423929d22628464856db1dd"
Oct 30 03:40:52.246428 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.246132    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc627a37430812e902275cf518392edaadf57747b423929d22628464856db1dd"} err="failed to get container status \"bc627a37430812e902275cf518392edaadf57747b423929d22628464856db1dd\": rpc error: code = NotFound desc = could not find container \"bc627a37430812e902275cf518392edaadf57747b423929d22628464856db1dd\": container with ID starting with bc627a37430812e902275cf518392edaadf57747b423929d22628464856db1dd not found: ID does not exist"
Oct 30 03:40:52.250548 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.250530    4960 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-web-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:52.250596 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.250550    4960 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-cluster-tls-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:52.250596 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.250559    4960 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1b42492-cd77-4565-913b-94670d1be346-tls-assets\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:52.250596 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.250568    4960 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1b42492-cd77-4565-913b-94670d1be346-config-out\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:52.250596 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.250576    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzx5h\" (UniqueName: \"kubernetes.io/projected/d1b42492-cd77-4565-913b-94670d1be346-kube-api-access-lzx5h\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:52.250596 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.250584    4960 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1b42492-cd77-4565-913b-94670d1be346-config-volume\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:40:52.276069 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.276045    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 03:40:52.281426 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:52.281408    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 03:40:53.502950 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:53.502914    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1b42492-cd77-4565-913b-94670d1be346" path="/var/lib/kubelet/pods/d1b42492-cd77-4565-913b-94670d1be346/volumes"
Oct 30 03:40:53.953625 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:40:53.953594    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:40:56.244946 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:40:56.244914    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:00.211480 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:00.211427    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24"
Oct 30 03:41:23.952797 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:41:23.952758    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:41:39.483131 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:39.483095    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-mxg24"]
Oct 30 03:41:39.483513 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:39.483316    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24" podUID="595feb8a-8943-4722-a57c-40b9a4217941" containerName="korrel8r" containerID="cri-o://009c07115f3fc685743a592236bb8b6abea953eb694ea02e25324f9471b9e868" gracePeriod=30
Oct 30 03:41:39.725399 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:39.725375    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24"
Oct 30 03:41:39.777457 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:39.777386    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595feb8a-8943-4722-a57c-40b9a4217941-kube-api-access-r8lv6\" (UniqueName: \"kubernetes.io/projected/595feb8a-8943-4722-a57c-40b9a4217941-kube-api-access-r8lv6\") pod \"595feb8a-8943-4722-a57c-40b9a4217941\" (UID: \"595feb8a-8943-4722-a57c-40b9a4217941\") "
Oct 30 03:41:39.777457 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:39.777420    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/595feb8a-8943-4722-a57c-40b9a4217941-config\" (UniqueName: \"kubernetes.io/configmap/595feb8a-8943-4722-a57c-40b9a4217941-config\") pod \"595feb8a-8943-4722-a57c-40b9a4217941\" (UID: \"595feb8a-8943-4722-a57c-40b9a4217941\") "
Oct 30 03:41:39.777642 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:39.777473    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/595feb8a-8943-4722-a57c-40b9a4217941-tls\" (UniqueName: \"kubernetes.io/secret/595feb8a-8943-4722-a57c-40b9a4217941-tls\") pod \"595feb8a-8943-4722-a57c-40b9a4217941\" (UID: \"595feb8a-8943-4722-a57c-40b9a4217941\") "
Oct 30 03:41:39.777910 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:39.777878    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/595feb8a-8943-4722-a57c-40b9a4217941-config" pod "595feb8a-8943-4722-a57c-40b9a4217941" (UID: "595feb8a-8943-4722-a57c-40b9a4217941"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:39.779755 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:39.779725    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/595feb8a-8943-4722-a57c-40b9a4217941-tls" pod "595feb8a-8943-4722-a57c-40b9a4217941" (UID: "595feb8a-8943-4722-a57c-40b9a4217941"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:39.779905 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:39.779880    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595feb8a-8943-4722-a57c-40b9a4217941-kube-api-access-r8lv6" pod "595feb8a-8943-4722-a57c-40b9a4217941" (UID: "595feb8a-8943-4722-a57c-40b9a4217941"). InnerVolumeSpecName "kube-api-access-r8lv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:41:39.879074 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:39.879045    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/595feb8a-8943-4722-a57c-40b9a4217941-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:41:39.879074 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:39.879078    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8lv6\" (UniqueName: \"kubernetes.io/projected/595feb8a-8943-4722-a57c-40b9a4217941-kube-api-access-r8lv6\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:41:39.879268 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:39.879094    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/595feb8a-8943-4722-a57c-40b9a4217941-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:41:39.894559 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:39.894535    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:41:39.894808 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:39.894781    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="fb86fdf7-abf2-4ab5-aa14-6a5e98563455" containerName="otc-container" containerID="cri-o://315072835b14a8da0095223bb606919b2f2ad10aafc28228ec044a4ec930ed68" gracePeriod=30
Oct 30 03:41:39.967524 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:39.967491    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:40.079226 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.079193    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:40.134479 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.134457    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:40.181312 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.181287    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-otc-internal\" (UniqueName: \"kubernetes.io/configmap/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-otc-internal\") pod \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\" (UID: \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\") "
Oct 30 03:41:40.181457 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.181355    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-otlp-tls-certs\") pod \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\" (UID: \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\") "
Oct 30 03:41:40.181457 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.181385    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-kube-api-access-vrkmr\" (UniqueName: \"kubernetes.io/projected/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-kube-api-access-vrkmr\") pod \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\" (UID: \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\") "
Oct 30 03:41:40.181457 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.181415    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-monitoring-tls-certs\") pod \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\" (UID: \"fb86fdf7-abf2-4ab5-aa14-6a5e98563455\") "
Oct 30 03:41:40.183425 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.181924    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-otc-internal" pod "fb86fdf7-abf2-4ab5-aa14-6a5e98563455" (UID: "fb86fdf7-abf2-4ab5-aa14-6a5e98563455"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:40.184255 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.184174    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-otlp-tls-certs" pod "fb86fdf7-abf2-4ab5-aa14-6a5e98563455" (UID: "fb86fdf7-abf2-4ab5-aa14-6a5e98563455"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:40.184365 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.184349    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-monitoring-tls-certs" pod "fb86fdf7-abf2-4ab5-aa14-6a5e98563455" (UID: "fb86fdf7-abf2-4ab5-aa14-6a5e98563455"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:40.184975 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.184955    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-kube-api-access-vrkmr" pod "fb86fdf7-abf2-4ab5-aa14-6a5e98563455" (UID: "fb86fdf7-abf2-4ab5-aa14-6a5e98563455"). InnerVolumeSpecName "kube-api-access-vrkmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:41:40.214901 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.214870    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:40.282231 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.282210    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:41:40.282231 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.282230    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrkmr\" (UniqueName: \"kubernetes.io/projected/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-kube-api-access-vrkmr\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:41:40.282368 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.282241    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:41:40.282368 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.282251    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fb86fdf7-abf2-4ab5-aa14-6a5e98563455-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:41:40.443533 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.443367    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:40.443862 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.443829    4960 generic.go:356] "Generic (PLEG): container finished" podID="fb86fdf7-abf2-4ab5-aa14-6a5e98563455" containerID="315072835b14a8da0095223bb606919b2f2ad10aafc28228ec044a4ec930ed68" exitCode=0
Oct 30 03:41:40.443986 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.443926    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fb86fdf7-abf2-4ab5-aa14-6a5e98563455","Type":"ContainerDied","Data":"315072835b14a8da0095223bb606919b2f2ad10aafc28228ec044a4ec930ed68"}
Oct 30 03:41:40.444099 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.443973    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:40.444099 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.444032    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fb86fdf7-abf2-4ab5-aa14-6a5e98563455","Type":"ContainerDied","Data":"5998330fd70f5e0c07fe25f8c93b05dadf3e86bf4e33791aeadea6171eb9816b"}
Oct 30 03:41:40.444099 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.444080    4960 scope.go:122] "RemoveContainer" containerID="315072835b14a8da0095223bb606919b2f2ad10aafc28228ec044a4ec930ed68"
Oct 30 03:41:40.445434 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.445265    4960 generic.go:356] "Generic (PLEG): container finished" podID="595feb8a-8943-4722-a57c-40b9a4217941" containerID="009c07115f3fc685743a592236bb8b6abea953eb694ea02e25324f9471b9e868" exitCode=0
Oct 30 03:41:40.445434 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.445313    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24" event={"ID":"595feb8a-8943-4722-a57c-40b9a4217941","Type":"ContainerDied","Data":"009c07115f3fc685743a592236bb8b6abea953eb694ea02e25324f9471b9e868"}
Oct 30 03:41:40.445434 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.445331    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24"
Oct 30 03:41:40.445434 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.445386    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-mxg24" event={"ID":"595feb8a-8943-4722-a57c-40b9a4217941","Type":"ContainerDied","Data":"5db71151de6846793ebea20d8971a52efbf4c509733145f0fb486fc367c77e6f"}
Oct 30 03:41:40.457118 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.457099    4960 scope.go:122] "RemoveContainer" containerID="315072835b14a8da0095223bb606919b2f2ad10aafc28228ec044a4ec930ed68"
Oct 30 03:41:40.459516 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:41:40.459488    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"315072835b14a8da0095223bb606919b2f2ad10aafc28228ec044a4ec930ed68\": container with ID starting with 315072835b14a8da0095223bb606919b2f2ad10aafc28228ec044a4ec930ed68 not found: ID does not exist" containerID="315072835b14a8da0095223bb606919b2f2ad10aafc28228ec044a4ec930ed68"
Oct 30 03:41:40.459625 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.459526    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315072835b14a8da0095223bb606919b2f2ad10aafc28228ec044a4ec930ed68"} err="failed to get container status \"315072835b14a8da0095223bb606919b2f2ad10aafc28228ec044a4ec930ed68\": rpc error: code = NotFound desc = could not find container \"315072835b14a8da0095223bb606919b2f2ad10aafc28228ec044a4ec930ed68\": container with ID starting with 315072835b14a8da0095223bb606919b2f2ad10aafc28228ec044a4ec930ed68 not found: ID does not exist"
Oct 30 03:41:40.459625 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.459550    4960 scope.go:122] "RemoveContainer" containerID="009c07115f3fc685743a592236bb8b6abea953eb694ea02e25324f9471b9e868"
Oct 30 03:41:40.468039 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.468020    4960 scope.go:122] "RemoveContainer" containerID="009c07115f3fc685743a592236bb8b6abea953eb694ea02e25324f9471b9e868"
Oct 30 03:41:40.468345 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:41:40.468321    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"009c07115f3fc685743a592236bb8b6abea953eb694ea02e25324f9471b9e868\": container with ID starting with 009c07115f3fc685743a592236bb8b6abea953eb694ea02e25324f9471b9e868 not found: ID does not exist" containerID="009c07115f3fc685743a592236bb8b6abea953eb694ea02e25324f9471b9e868"
Oct 30 03:41:40.468454 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.468350    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"009c07115f3fc685743a592236bb8b6abea953eb694ea02e25324f9471b9e868"} err="failed to get container status \"009c07115f3fc685743a592236bb8b6abea953eb694ea02e25324f9471b9e868\": rpc error: code = NotFound desc = could not find container \"009c07115f3fc685743a592236bb8b6abea953eb694ea02e25324f9471b9e868\": container with ID starting with 009c07115f3fc685743a592236bb8b6abea953eb694ea02e25324f9471b9e868 not found: ID does not exist"
Oct 30 03:41:40.499005 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.498985    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-mxg24"]
Oct 30 03:41:40.500873 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.500851    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-mxg24"]
Oct 30 03:41:40.539774 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.539753    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:41:40.544422 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.544403    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:41:40.566748 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:40.566723    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:41.501948 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:41.501916    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="595feb8a-8943-4722-a57c-40b9a4217941" path="/var/lib/kubelet/pods/595feb8a-8943-4722-a57c-40b9a4217941/volumes"
Oct 30 03:41:41.502351 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:41.502295    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb86fdf7-abf2-4ab5-aa14-6a5e98563455" path="/var/lib/kubelet/pods/fb86fdf7-abf2-4ab5-aa14-6a5e98563455/volumes"
Oct 30 03:41:43.524608 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.524571    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:43.587378 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.587350    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-b42nv"]
Oct 30 03:41:43.587687 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.587674    4960 cpu_manager.go:414] "containerMap: removing container" podUID="595feb8a-8943-4722-a57c-40b9a4217941" containerName="korrel8r"
Oct 30 03:41:43.587737 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.587689    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="595feb8a-8943-4722-a57c-40b9a4217941" containerName="korrel8r"
Oct 30 03:41:43.587737 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.587702    4960 cpu_manager.go:414] "containerMap: removing container" podUID="fb86fdf7-abf2-4ab5-aa14-6a5e98563455" containerName="otc-container"
Oct 30 03:41:43.587737 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.587708    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb86fdf7-abf2-4ab5-aa14-6a5e98563455" containerName="otc-container"
Oct 30 03:41:43.587737 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.587715    4960 cpu_manager.go:414] "containerMap: removing container" podUID="d1b42492-cd77-4565-913b-94670d1be346" containerName="init-config-reloader"
Oct 30 03:41:43.587737 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.587724    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1b42492-cd77-4565-913b-94670d1be346" containerName="init-config-reloader"
Oct 30 03:41:43.587886 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.587744    4960 cpu_manager.go:414] "containerMap: removing container" podUID="d1b42492-cd77-4565-913b-94670d1be346" containerName="alertmanager"
Oct 30 03:41:43.587886 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.587748    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1b42492-cd77-4565-913b-94670d1be346" containerName="alertmanager"
Oct 30 03:41:43.587886 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.587755    4960 cpu_manager.go:414] "containerMap: removing container" podUID="d1b42492-cd77-4565-913b-94670d1be346" containerName="config-reloader"
Oct 30 03:41:43.587886 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.587760    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1b42492-cd77-4565-913b-94670d1be346" containerName="config-reloader"
Oct 30 03:41:43.587886 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.587829    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1b42492-cd77-4565-913b-94670d1be346" containerName="alertmanager"
Oct 30 03:41:43.587886 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.587837    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1b42492-cd77-4565-913b-94670d1be346" containerName="config-reloader"
Oct 30 03:41:43.587886 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.587846    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="595feb8a-8943-4722-a57c-40b9a4217941" containerName="korrel8r"
Oct 30 03:41:43.587886 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.587852    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb86fdf7-abf2-4ab5-aa14-6a5e98563455" containerName="otc-container"
Oct 30 03:41:43.592216 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.592200    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv"
Oct 30 03:41:43.625465 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.625438    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-vfhz7\""
Oct 30 03:41:43.625629 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.625492    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\""
Oct 30 03:41:43.625629 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.625539    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:41:43.625752 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.625679    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:41:43.625752 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.625706    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\""
Oct 30 03:41:43.627242 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.627222    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-b42nv"]
Oct 30 03:41:43.663007 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.662984    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:43.707917 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.707893    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/573cfada-0c9f-4a97-8023-ffcf656d96cb-config\") pod \"korrel8r-6644769df5-b42nv\" (UID: \"573cfada-0c9f-4a97-8023-ffcf656d96cb\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv"
Oct 30 03:41:43.708013 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.707926    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/573cfada-0c9f-4a97-8023-ffcf656d96cb-tls\") pod \"korrel8r-6644769df5-b42nv\" (UID: \"573cfada-0c9f-4a97-8023-ffcf656d96cb\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv"
Oct 30 03:41:43.708013 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.707944    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrsfs\" (UniqueName: \"kubernetes.io/projected/573cfada-0c9f-4a97-8023-ffcf656d96cb-kube-api-access-rrsfs\") pod \"korrel8r-6644769df5-b42nv\" (UID: \"573cfada-0c9f-4a97-8023-ffcf656d96cb\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv"
Oct 30 03:41:43.783805 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.783729    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:43.808781 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.808755    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/573cfada-0c9f-4a97-8023-ffcf656d96cb-config\") pod \"korrel8r-6644769df5-b42nv\" (UID: \"573cfada-0c9f-4a97-8023-ffcf656d96cb\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv"
Oct 30 03:41:43.808879 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.808792    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/573cfada-0c9f-4a97-8023-ffcf656d96cb-tls\") pod \"korrel8r-6644769df5-b42nv\" (UID: \"573cfada-0c9f-4a97-8023-ffcf656d96cb\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv"
Oct 30 03:41:43.808879 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.808810    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrsfs\" (UniqueName: \"kubernetes.io/projected/573cfada-0c9f-4a97-8023-ffcf656d96cb-kube-api-access-rrsfs\") pod \"korrel8r-6644769df5-b42nv\" (UID: \"573cfada-0c9f-4a97-8023-ffcf656d96cb\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv"
Oct 30 03:41:43.809493 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.809473    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/573cfada-0c9f-4a97-8023-ffcf656d96cb-config\") pod \"korrel8r-6644769df5-b42nv\" (UID: \"573cfada-0c9f-4a97-8023-ffcf656d96cb\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv"
Oct 30 03:41:43.811400 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.811372    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/573cfada-0c9f-4a97-8023-ffcf656d96cb-tls\") pod \"korrel8r-6644769df5-b42nv\" (UID: \"573cfada-0c9f-4a97-8023-ffcf656d96cb\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv"
Oct 30 03:41:43.829824 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.829800    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrsfs\" (UniqueName: \"kubernetes.io/projected/573cfada-0c9f-4a97-8023-ffcf656d96cb-kube-api-access-rrsfs\") pod \"korrel8r-6644769df5-b42nv\" (UID: \"573cfada-0c9f-4a97-8023-ffcf656d96cb\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv"
Oct 30 03:41:43.904305 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:43.904278    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv"
Oct 30 03:41:44.198533 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.198131    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-b42nv"]
Oct 30 03:41:44.200391 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:41:44.200361    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573cfada_0c9f_4a97_8023_ffcf656d96cb.slice/crio-19d71adefa3d766ceb436e4ad06ff056e2017a9dc9eb37cbbbce8bb6fe750b0d WatchSource:0}: Error finding container 19d71adefa3d766ceb436e4ad06ff056e2017a9dc9eb37cbbbce8bb6fe750b0d: Status 404 returned error can't find the container with id 19d71adefa3d766ceb436e4ad06ff056e2017a9dc9eb37cbbbce8bb6fe750b0d
Oct 30 03:41:44.209969 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.209947    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:41:44.214648 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.214627    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:44.236827 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.236800    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:41:44.264332 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.264296    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:44.303883 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.303857    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:41:44.304035 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.303938    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:41:44.304035 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.303990    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-47hhb\""
Oct 30 03:41:44.306282 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.306262    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\""
Oct 30 03:41:44.314354 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.314331    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:44.314461 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.314404    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5dzx\" (UniqueName: \"kubernetes.io/projected/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-kube-api-access-v5dzx\") pod \"data-science-collector-collector-1\" (UID: \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:44.314501 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.314462    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:44.314501 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.314488    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:44.406577 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.406554    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:44.415679 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.415653    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5dzx\" (UniqueName: \"kubernetes.io/projected/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-kube-api-access-v5dzx\") pod \"data-science-collector-collector-1\" (UID: \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:44.415784 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.415686    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:44.415784 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.415707    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:44.415784 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.415744    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:44.416442 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.416410    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:44.418373 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.418353    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:44.418373 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.418364    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:44.436662 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.436635    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5dzx\" (UniqueName: \"kubernetes.io/projected/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-kube-api-access-v5dzx\") pod \"data-science-collector-collector-1\" (UID: \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:44.466996 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.466935    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv" event={"ID":"573cfada-0c9f-4a97-8023-ffcf656d96cb","Type":"ContainerStarted","Data":"d937f9ce4f9f70b703158170de2197581277297e95b1f1c3a30b9ef62466b197"}
Oct 30 03:41:44.467099 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.466995    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv" event={"ID":"573cfada-0c9f-4a97-8023-ffcf656d96cb","Type":"ContainerStarted","Data":"19d71adefa3d766ceb436e4ad06ff056e2017a9dc9eb37cbbbce8bb6fe750b0d"}
Oct 30 03:41:44.467099 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.467039    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv"
Oct 30 03:41:44.513263 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.513225    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv" podStartSLOduration=1.5132138400000001 podStartE2EDuration="1.51321384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:44.511739988 +0000 UTC m=+3585.160788410" watchObservedRunningTime="2026-10-30 03:41:44.51321384 +0000 UTC m=+3585.162262261"
Oct 30 03:41:44.541969 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.541942    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:44.549014 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.548996    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:44.853207 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.853159    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:41:44.856396 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:41:44.856360    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2e6b30_4a46_4e76_98ed_21ef4a369f73.slice/crio-e027d554f9444205f4767e478d87b00950bdd5111a7101789400e7378bc84832 WatchSource:0}: Error finding container e027d554f9444205f4767e478d87b00950bdd5111a7101789400e7378bc84832: Status 404 returned error can't find the container with id e027d554f9444205f4767e478d87b00950bdd5111a7101789400e7378bc84832
Oct 30 03:41:44.858690 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.858653    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:41:44.858806 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:44.858729    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:41:45.472513 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:45.472479    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2e2e6b30-4a46-4e76-98ed-21ef4a369f73","Type":"ContainerStarted","Data":"878c94c8c21433f4d661afa8af6fd8f4394e5bdb61702d962bb45dac3966f2ed"}
Oct 30 03:41:45.472686 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:45.472552    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2e2e6b30-4a46-4e76-98ed-21ef4a369f73","Type":"ContainerStarted","Data":"e027d554f9444205f4767e478d87b00950bdd5111a7101789400e7378bc84832"}
Oct 30 03:41:45.517464 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:45.517412    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.517398265 podStartE2EDuration="1.517398265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:45.515850874 +0000 UTC m=+3586.164899295" watchObservedRunningTime="2026-10-30 03:41:45.517398265 +0000 UTC m=+3586.166446687"
Oct 30 03:41:45.812639 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:45.812566    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:45.957558 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:45.957527    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:53.952447 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:41:53.952405    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:41:54.827881 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:54.827849    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:54.932236 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:54.932207    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:55.475106 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:55.475075    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv"
Oct 30 03:41:56.368766 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:56.368732    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:58.963558 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:58.963527    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-b42nv"]
Oct 30 03:41:58.963972 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:58.963742    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv" podUID="573cfada-0c9f-4a97-8023-ffcf656d96cb" containerName="korrel8r" containerID="cri-o://d937f9ce4f9f70b703158170de2197581277297e95b1f1c3a30b9ef62466b197" gracePeriod=30
Oct 30 03:41:59.214402 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.214340    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv"
Oct 30 03:41:59.327191 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.327160    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/573cfada-0c9f-4a97-8023-ffcf656d96cb-kube-api-access-rrsfs\" (UniqueName: \"kubernetes.io/projected/573cfada-0c9f-4a97-8023-ffcf656d96cb-kube-api-access-rrsfs\") pod \"573cfada-0c9f-4a97-8023-ffcf656d96cb\" (UID: \"573cfada-0c9f-4a97-8023-ffcf656d96cb\") "
Oct 30 03:41:59.327366 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.327224    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/573cfada-0c9f-4a97-8023-ffcf656d96cb-config\" (UniqueName: \"kubernetes.io/configmap/573cfada-0c9f-4a97-8023-ffcf656d96cb-config\") pod \"573cfada-0c9f-4a97-8023-ffcf656d96cb\" (UID: \"573cfada-0c9f-4a97-8023-ffcf656d96cb\") "
Oct 30 03:41:59.327366 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.327282    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/573cfada-0c9f-4a97-8023-ffcf656d96cb-tls\" (UniqueName: \"kubernetes.io/secret/573cfada-0c9f-4a97-8023-ffcf656d96cb-tls\") pod \"573cfada-0c9f-4a97-8023-ffcf656d96cb\" (UID: \"573cfada-0c9f-4a97-8023-ffcf656d96cb\") "
Oct 30 03:41:59.327594 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.327561    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573cfada-0c9f-4a97-8023-ffcf656d96cb-config" pod "573cfada-0c9f-4a97-8023-ffcf656d96cb" (UID: "573cfada-0c9f-4a97-8023-ffcf656d96cb"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:59.329408 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.329380    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573cfada-0c9f-4a97-8023-ffcf656d96cb-kube-api-access-rrsfs" pod "573cfada-0c9f-4a97-8023-ffcf656d96cb" (UID: "573cfada-0c9f-4a97-8023-ffcf656d96cb"). InnerVolumeSpecName "kube-api-access-rrsfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:41:59.329515 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.329389    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573cfada-0c9f-4a97-8023-ffcf656d96cb-tls" pod "573cfada-0c9f-4a97-8023-ffcf656d96cb" (UID: "573cfada-0c9f-4a97-8023-ffcf656d96cb"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:59.370455 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.370423    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:41:59.370685 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.370657    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="2e2e6b30-4a46-4e76-98ed-21ef4a369f73" containerName="otc-container" containerID="cri-o://878c94c8c21433f4d661afa8af6fd8f4394e5bdb61702d962bb45dac3966f2ed" gracePeriod=30
Oct 30 03:41:59.402564 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.402536    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:59.428679 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.428655    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrsfs\" (UniqueName: \"kubernetes.io/projected/573cfada-0c9f-4a97-8023-ffcf656d96cb-kube-api-access-rrsfs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:41:59.428679 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.428680    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/573cfada-0c9f-4a97-8023-ffcf656d96cb-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:41:59.429004 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.428693    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/573cfada-0c9f-4a97-8023-ffcf656d96cb-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:41:59.528928 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.528878    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:59.532231 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.532166    4960 generic.go:356] "Generic (PLEG): container finished" podID="2e2e6b30-4a46-4e76-98ed-21ef4a369f73" containerID="878c94c8c21433f4d661afa8af6fd8f4394e5bdb61702d962bb45dac3966f2ed" exitCode=0
Oct 30 03:41:59.532360 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.532230    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2e2e6b30-4a46-4e76-98ed-21ef4a369f73","Type":"ContainerDied","Data":"878c94c8c21433f4d661afa8af6fd8f4394e5bdb61702d962bb45dac3966f2ed"}
Oct 30 03:41:59.534090 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.534064    4960 generic.go:356] "Generic (PLEG): container finished" podID="573cfada-0c9f-4a97-8023-ffcf656d96cb" containerID="d937f9ce4f9f70b703158170de2197581277297e95b1f1c3a30b9ef62466b197" exitCode=0
Oct 30 03:41:59.534236 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.534141    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv" event={"ID":"573cfada-0c9f-4a97-8023-ffcf656d96cb","Type":"ContainerDied","Data":"d937f9ce4f9f70b703158170de2197581277297e95b1f1c3a30b9ef62466b197"}
Oct 30 03:41:59.534236 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.534218    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv" event={"ID":"573cfada-0c9f-4a97-8023-ffcf656d96cb","Type":"ContainerDied","Data":"19d71adefa3d766ceb436e4ad06ff056e2017a9dc9eb37cbbbce8bb6fe750b0d"}
Oct 30 03:41:59.534374 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.534249    4960 scope.go:122] "RemoveContainer" containerID="d937f9ce4f9f70b703158170de2197581277297e95b1f1c3a30b9ef62466b197"
Oct 30 03:41:59.534458 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.534447    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-b42nv"
Oct 30 03:41:59.545694 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.545671    4960 scope.go:122] "RemoveContainer" containerID="d937f9ce4f9f70b703158170de2197581277297e95b1f1c3a30b9ef62466b197"
Oct 30 03:41:59.546162 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:41:59.546138    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d937f9ce4f9f70b703158170de2197581277297e95b1f1c3a30b9ef62466b197\": container with ID starting with d937f9ce4f9f70b703158170de2197581277297e95b1f1c3a30b9ef62466b197 not found: ID does not exist" containerID="d937f9ce4f9f70b703158170de2197581277297e95b1f1c3a30b9ef62466b197"
Oct 30 03:41:59.546269 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.546169    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d937f9ce4f9f70b703158170de2197581277297e95b1f1c3a30b9ef62466b197"} err="failed to get container status \"d937f9ce4f9f70b703158170de2197581277297e95b1f1c3a30b9ef62466b197\": rpc error: code = NotFound desc = could not find container \"d937f9ce4f9f70b703158170de2197581277297e95b1f1c3a30b9ef62466b197\": container with ID starting with d937f9ce4f9f70b703158170de2197581277297e95b1f1c3a30b9ef62466b197 not found: ID does not exist"
Oct 30 03:41:59.588583 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.588560    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-b42nv"]
Oct 30 03:41:59.592985 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.592963    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-b42nv"]
Oct 30 03:41:59.630688 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.630666    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:59.651690 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.651668    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:59.730792 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.730748    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-monitoring-tls-certs\") pod \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\" (UID: \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\") "
Oct 30 03:41:59.730960 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.730847    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-kube-api-access-v5dzx\" (UniqueName: \"kubernetes.io/projected/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-kube-api-access-v5dzx\") pod \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\" (UID: \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\") "
Oct 30 03:41:59.730960 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.730875    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-otlp-tls-certs\") pod \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\" (UID: \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\") "
Oct 30 03:41:59.730960 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.730920    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-otc-internal\") pod \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\" (UID: \"2e2e6b30-4a46-4e76-98ed-21ef4a369f73\") "
Oct 30 03:41:59.731308 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.731279    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-otc-internal" pod "2e2e6b30-4a46-4e76-98ed-21ef4a369f73" (UID: "2e2e6b30-4a46-4e76-98ed-21ef4a369f73"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:59.732975 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.732951    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-kube-api-access-v5dzx" pod "2e2e6b30-4a46-4e76-98ed-21ef4a369f73" (UID: "2e2e6b30-4a46-4e76-98ed-21ef4a369f73"). InnerVolumeSpecName "kube-api-access-v5dzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:41:59.732975 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.732967    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-monitoring-tls-certs" pod "2e2e6b30-4a46-4e76-98ed-21ef4a369f73" (UID: "2e2e6b30-4a46-4e76-98ed-21ef4a369f73"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:59.733119 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.732988    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-otlp-tls-certs" pod "2e2e6b30-4a46-4e76-98ed-21ef4a369f73" (UID: "2e2e6b30-4a46-4e76-98ed-21ef4a369f73"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:59.831459 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.831432    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5dzx\" (UniqueName: \"kubernetes.io/projected/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-kube-api-access-v5dzx\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:41:59.831459 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.831459    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:41:59.831675 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.831469    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:41:59.831675 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.831499    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2e6b30-4a46-4e76-98ed-21ef4a369f73-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:41:59.848396 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.846222    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:41:59.850484 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.850461    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:41:59.854344 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.854307    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:41:59.862775 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.862746    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:41:59.915293 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:41:59.915261    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:00.042611 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:00.042580    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:00.539301 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:00.539274    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:00.539464 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:00.539269    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2e2e6b30-4a46-4e76-98ed-21ef4a369f73","Type":"ContainerDied","Data":"e027d554f9444205f4767e478d87b00950bdd5111a7101789400e7378bc84832"}
Oct 30 03:42:00.539464 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:00.539411    4960 scope.go:122] "RemoveContainer" containerID="878c94c8c21433f4d661afa8af6fd8f4394e5bdb61702d962bb45dac3966f2ed"
Oct 30 03:42:00.593317 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:00.593287    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:42:00.606981 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:00.606953    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:42:01.503085 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:01.503049    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e2e6b30-4a46-4e76-98ed-21ef4a369f73" path="/var/lib/kubelet/pods/2e2e6b30-4a46-4e76-98ed-21ef4a369f73/volumes"
Oct 30 03:42:01.503501 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:01.503449    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="573cfada-0c9f-4a97-8023-ffcf656d96cb" path="/var/lib/kubelet/pods/573cfada-0c9f-4a97-8023-ffcf656d96cb/volumes"
Oct 30 03:42:03.600956 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.600919    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:03.626902 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.626870    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-4pzml"]
Oct 30 03:42:03.627217 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.627204    4960 cpu_manager.go:414] "containerMap: removing container" podUID="2e2e6b30-4a46-4e76-98ed-21ef4a369f73" containerName="otc-container"
Oct 30 03:42:03.627266 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.627220    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2e6b30-4a46-4e76-98ed-21ef4a369f73" containerName="otc-container"
Oct 30 03:42:03.627266 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.627251    4960 cpu_manager.go:414] "containerMap: removing container" podUID="573cfada-0c9f-4a97-8023-ffcf656d96cb" containerName="korrel8r"
Oct 30 03:42:03.627266 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.627256    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="573cfada-0c9f-4a97-8023-ffcf656d96cb" containerName="korrel8r"
Oct 30 03:42:03.627354 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.627319    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2e6b30-4a46-4e76-98ed-21ef4a369f73" containerName="otc-container"
Oct 30 03:42:03.627354 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.627330    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="573cfada-0c9f-4a97-8023-ffcf656d96cb" containerName="korrel8r"
Oct 30 03:42:03.631762 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.631744    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml"
Oct 30 03:42:03.649812 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.649787    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-4pzml"]
Oct 30 03:42:03.662492 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.662466    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:42:03.662625 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.662500    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\""
Oct 30 03:42:03.662625 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.662511    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:42:03.662625 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.662562    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-lhvm5\""
Oct 30 03:42:03.662625 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.662576    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\""
Oct 30 03:42:03.724503 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.724475    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:03.763700 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.763677    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-tls\") pod \"korrel8r-6644769df5-4pzml\" (UID: \"45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml"
Oct 30 03:42:03.763800 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.763706    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zlzx\" (UniqueName: \"kubernetes.io/projected/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-kube-api-access-8zlzx\") pod \"korrel8r-6644769df5-4pzml\" (UID: \"45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml"
Oct 30 03:42:03.763800 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.763736    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-config\") pod \"korrel8r-6644769df5-4pzml\" (UID: \"45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml"
Oct 30 03:42:03.842439 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.842413    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:03.864503 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.864440    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-config\") pod \"korrel8r-6644769df5-4pzml\" (UID: \"45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml"
Oct 30 03:42:03.864603 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.864513    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-tls\") pod \"korrel8r-6644769df5-4pzml\" (UID: \"45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml"
Oct 30 03:42:03.864603 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.864548    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zlzx\" (UniqueName: \"kubernetes.io/projected/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-kube-api-access-8zlzx\") pod \"korrel8r-6644769df5-4pzml\" (UID: \"45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml"
Oct 30 03:42:03.865088 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.865071    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-config\") pod \"korrel8r-6644769df5-4pzml\" (UID: \"45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml"
Oct 30 03:42:03.867015 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.866998    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-tls\") pod \"korrel8r-6644769df5-4pzml\" (UID: \"45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml"
Oct 30 03:42:03.886362 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.886342    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zlzx\" (UniqueName: \"kubernetes.io/projected/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-kube-api-access-8zlzx\") pod \"korrel8r-6644769df5-4pzml\" (UID: \"45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml"
Oct 30 03:42:03.944503 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:03.944478    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml"
Oct 30 03:42:04.240444 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.240404    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-4pzml"]
Oct 30 03:42:04.242727 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:42:04.242684    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f61d6b_fd6d_43c7_ab4f_a36fc5fe2ff1.slice/crio-0ea15ffd73ee0a2a5ef8f613ab8d9ab7a84b0d6841fed80aa71f5f69f4f30c6d WatchSource:0}: Error finding container 0ea15ffd73ee0a2a5ef8f613ab8d9ab7a84b0d6841fed80aa71f5f69f4f30c6d: Status 404 returned error can't find the container with id 0ea15ffd73ee0a2a5ef8f613ab8d9ab7a84b0d6841fed80aa71f5f69f4f30c6d
Oct 30 03:42:04.296645 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.296616    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:42:04.301454 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.301431    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:04.322515 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.322491    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:42:04.335060 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.335042    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:42:04.335331 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.335313    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:42:04.335406 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.335350    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-mxwv2\""
Oct 30 03:42:04.341446 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.341419    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:04.370428 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.370404    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9vn4\" (UniqueName: \"kubernetes.io/projected/29798940-9dfd-4438-b3cf-c0992468ba16-kube-api-access-h9vn4\") pod \"data-science-collector-collector-1\" (UID: \"29798940-9dfd-4438-b3cf-c0992468ba16\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:04.370528 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.370455    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/29798940-9dfd-4438-b3cf-c0992468ba16-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"29798940-9dfd-4438-b3cf-c0992468ba16\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:04.370528 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.370493    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/29798940-9dfd-4438-b3cf-c0992468ba16-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"29798940-9dfd-4438-b3cf-c0992468ba16\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:04.370682 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.370555    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/29798940-9dfd-4438-b3cf-c0992468ba16-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"29798940-9dfd-4438-b3cf-c0992468ba16\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:04.387693 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.387674    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\""
Oct 30 03:42:04.471645 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.471618    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9vn4\" (UniqueName: \"kubernetes.io/projected/29798940-9dfd-4438-b3cf-c0992468ba16-kube-api-access-h9vn4\") pod \"data-science-collector-collector-1\" (UID: \"29798940-9dfd-4438-b3cf-c0992468ba16\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:04.471753 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.471656    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/29798940-9dfd-4438-b3cf-c0992468ba16-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"29798940-9dfd-4438-b3cf-c0992468ba16\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:04.471753 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.471675    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/29798940-9dfd-4438-b3cf-c0992468ba16-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"29798940-9dfd-4438-b3cf-c0992468ba16\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:04.471753 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.471693    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/29798940-9dfd-4438-b3cf-c0992468ba16-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"29798940-9dfd-4438-b3cf-c0992468ba16\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:04.472435 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.472416    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/29798940-9dfd-4438-b3cf-c0992468ba16-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"29798940-9dfd-4438-b3cf-c0992468ba16\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:04.474118 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.474100    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/29798940-9dfd-4438-b3cf-c0992468ba16-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"29798940-9dfd-4438-b3cf-c0992468ba16\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:04.474209 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.474193    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/29798940-9dfd-4438-b3cf-c0992468ba16-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"29798940-9dfd-4438-b3cf-c0992468ba16\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:04.498308 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.498243    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9vn4\" (UniqueName: \"kubernetes.io/projected/29798940-9dfd-4438-b3cf-c0992468ba16-kube-api-access-h9vn4\") pod \"data-science-collector-collector-1\" (UID: \"29798940-9dfd-4438-b3cf-c0992468ba16\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:04.531156 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.531135    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:04.559148 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.559126    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml" event={"ID":"45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1","Type":"ContainerStarted","Data":"a8cdaa8570554a3e7988451326a7e52f33c1c0394c0cf0edcae2ba5c9d252f23"}
Oct 30 03:42:04.559274 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.559170    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml" event={"ID":"45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1","Type":"ContainerStarted","Data":"0ea15ffd73ee0a2a5ef8f613ab8d9ab7a84b0d6841fed80aa71f5f69f4f30c6d"}
Oct 30 03:42:04.559274 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.559225    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml"
Oct 30 03:42:04.607010 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.606972    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml" podStartSLOduration=1.606960533 podStartE2EDuration="1.606960533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:04.605869854 +0000 UTC m=+3605.254918276" watchObservedRunningTime="2026-10-30 03:42:04.606960533 +0000 UTC m=+3605.256008954"
Oct 30 03:42:04.616038 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.616016    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:04.926376 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.926353    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:42:04.927951 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:42:04.927925    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29798940_9dfd_4438_b3cf_c0992468ba16.slice/crio-511a2bb8797dd748152fa84ec620b26d50017ab09faf6db58edf5f5a1f541d83 WatchSource:0}: Error finding container 511a2bb8797dd748152fa84ec620b26d50017ab09faf6db58edf5f5a1f541d83: Status 404 returned error can't find the container with id 511a2bb8797dd748152fa84ec620b26d50017ab09faf6db58edf5f5a1f541d83
Oct 30 03:42:04.930102 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.930070    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:42:04.930214 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:04.930130    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:42:05.176748 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:05.176655    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:05.444373 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:05.444300    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:05.564998 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:05.564964    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"29798940-9dfd-4438-b3cf-c0992468ba16","Type":"ContainerStarted","Data":"67fccd74e6c51ed68503a59c82adcb319ba254d10047130460a1e7edca3ed154"}
Oct 30 03:42:05.565148 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:05.565008    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"29798940-9dfd-4438-b3cf-c0992468ba16","Type":"ContainerStarted","Data":"511a2bb8797dd748152fa84ec620b26d50017ab09faf6db58edf5f5a1f541d83"}
Oct 30 03:42:05.612438 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:05.612377    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.612361998 podStartE2EDuration="1.612361998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:05.611383123 +0000 UTC m=+3606.260431543" watchObservedRunningTime="2026-10-30 03:42:05.612361998 +0000 UTC m=+3606.261410410"
Oct 30 03:42:15.567201 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:15.567145    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml"
Oct 30 03:42:23.955778 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:42:23.955737    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:42:38.553543 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:38.553492    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:38.660493 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:38.660455    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:53.953475 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:42:53.953440    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:42:56.511872 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:42:56.511839    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:01.849300 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:01.849267    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-4pzml"]
Oct 30 03:43:01.849731 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:01.849446    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml" podUID="45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1" containerName="korrel8r" containerID="cri-o://a8cdaa8570554a3e7988451326a7e52f33c1c0394c0cf0edcae2ba5c9d252f23" gracePeriod=30
Oct 30 03:43:02.089861 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.089833    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml"
Oct 30 03:43:02.181437 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.181363    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-config\" (UniqueName: \"kubernetes.io/configmap/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-config\") pod \"45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1\" (UID: \"45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1\") "
Oct 30 03:43:02.181437 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.181403    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-tls\" (UniqueName: \"kubernetes.io/secret/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-tls\") pod \"45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1\" (UID: \"45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1\") "
Oct 30 03:43:02.181437 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.181427    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-kube-api-access-8zlzx\" (UniqueName: \"kubernetes.io/projected/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-kube-api-access-8zlzx\") pod \"45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1\" (UID: \"45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1\") "
Oct 30 03:43:02.182315 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.182030    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-config" pod "45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1" (UID: "45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:02.186226 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.184732    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-kube-api-access-8zlzx" pod "45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1" (UID: "45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1"). InnerVolumeSpecName "kube-api-access-8zlzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:02.189796 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.189643    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-tls" pod "45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1" (UID: "45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:02.254711 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.254681    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:02.254995 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.254964    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="29798940-9dfd-4438-b3cf-c0992468ba16" containerName="otc-container" containerID="cri-o://67fccd74e6c51ed68503a59c82adcb319ba254d10047130460a1e7edca3ed154" gracePeriod=30
Oct 30 03:43:02.283283 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.283256    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:43:02.283283 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.283284    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:43:02.283459 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.283295    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zlzx\" (UniqueName: \"kubernetes.io/projected/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1-kube-api-access-8zlzx\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:43:02.318332 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.318298    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:02.456090 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.455997    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:02.491734 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.491709    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:02.584891 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.584859    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29798940-9dfd-4438-b3cf-c0992468ba16-otc-internal\" (UniqueName: \"kubernetes.io/configmap/29798940-9dfd-4438-b3cf-c0992468ba16-otc-internal\") pod \"29798940-9dfd-4438-b3cf-c0992468ba16\" (UID: \"29798940-9dfd-4438-b3cf-c0992468ba16\") "
Oct 30 03:43:02.585066 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.584911    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29798940-9dfd-4438-b3cf-c0992468ba16-kube-api-access-h9vn4\" (UniqueName: \"kubernetes.io/projected/29798940-9dfd-4438-b3cf-c0992468ba16-kube-api-access-h9vn4\") pod \"29798940-9dfd-4438-b3cf-c0992468ba16\" (UID: \"29798940-9dfd-4438-b3cf-c0992468ba16\") "
Oct 30 03:43:02.585066 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.584953    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29798940-9dfd-4438-b3cf-c0992468ba16-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/29798940-9dfd-4438-b3cf-c0992468ba16-monitoring-tls-certs\") pod \"29798940-9dfd-4438-b3cf-c0992468ba16\" (UID: \"29798940-9dfd-4438-b3cf-c0992468ba16\") "
Oct 30 03:43:02.585066 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.585001    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29798940-9dfd-4438-b3cf-c0992468ba16-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/29798940-9dfd-4438-b3cf-c0992468ba16-otlp-tls-certs\") pod \"29798940-9dfd-4438-b3cf-c0992468ba16\" (UID: \"29798940-9dfd-4438-b3cf-c0992468ba16\") "
Oct 30 03:43:02.585592 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.585311    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29798940-9dfd-4438-b3cf-c0992468ba16-otc-internal" pod "29798940-9dfd-4438-b3cf-c0992468ba16" (UID: "29798940-9dfd-4438-b3cf-c0992468ba16"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:02.585714 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.585671    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/29798940-9dfd-4438-b3cf-c0992468ba16-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:43:02.587507 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.587478    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29798940-9dfd-4438-b3cf-c0992468ba16-otlp-tls-certs" pod "29798940-9dfd-4438-b3cf-c0992468ba16" (UID: "29798940-9dfd-4438-b3cf-c0992468ba16"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:02.587781 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.587760    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29798940-9dfd-4438-b3cf-c0992468ba16-monitoring-tls-certs" pod "29798940-9dfd-4438-b3cf-c0992468ba16" (UID: "29798940-9dfd-4438-b3cf-c0992468ba16"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:02.587856 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.587784    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29798940-9dfd-4438-b3cf-c0992468ba16-kube-api-access-h9vn4" pod "29798940-9dfd-4438-b3cf-c0992468ba16" (UID: "29798940-9dfd-4438-b3cf-c0992468ba16"). InnerVolumeSpecName "kube-api-access-h9vn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:02.686402 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.686377    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/29798940-9dfd-4438-b3cf-c0992468ba16-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:43:02.686402 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.686400    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9vn4\" (UniqueName: \"kubernetes.io/projected/29798940-9dfd-4438-b3cf-c0992468ba16-kube-api-access-h9vn4\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:43:02.686543 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.686411    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/29798940-9dfd-4438-b3cf-c0992468ba16-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:43:02.803764 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.803667    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:02.816623 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.816588    4960 generic.go:356] "Generic (PLEG): container finished" podID="29798940-9dfd-4438-b3cf-c0992468ba16" containerID="67fccd74e6c51ed68503a59c82adcb319ba254d10047130460a1e7edca3ed154" exitCode=0
Oct 30 03:43:02.816793 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.816634    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"29798940-9dfd-4438-b3cf-c0992468ba16","Type":"ContainerDied","Data":"67fccd74e6c51ed68503a59c82adcb319ba254d10047130460a1e7edca3ed154"}
Oct 30 03:43:02.816793 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.816671    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:02.816793 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.816706    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"29798940-9dfd-4438-b3cf-c0992468ba16","Type":"ContainerDied","Data":"511a2bb8797dd748152fa84ec620b26d50017ab09faf6db58edf5f5a1f541d83"}
Oct 30 03:43:02.816793 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.816737    4960 scope.go:122] "RemoveContainer" containerID="67fccd74e6c51ed68503a59c82adcb319ba254d10047130460a1e7edca3ed154"
Oct 30 03:43:02.819092 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.818926    4960 generic.go:356] "Generic (PLEG): container finished" podID="45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1" containerID="a8cdaa8570554a3e7988451326a7e52f33c1c0394c0cf0edcae2ba5c9d252f23" exitCode=0
Oct 30 03:43:02.819092 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.819055    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml" event={"ID":"45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1","Type":"ContainerDied","Data":"a8cdaa8570554a3e7988451326a7e52f33c1c0394c0cf0edcae2ba5c9d252f23"}
Oct 30 03:43:02.819290 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.819249    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml"
Oct 30 03:43:02.819290 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.819219    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-4pzml" event={"ID":"45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1","Type":"ContainerDied","Data":"0ea15ffd73ee0a2a5ef8f613ab8d9ab7a84b0d6841fed80aa71f5f69f4f30c6d"}
Oct 30 03:43:02.827809 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.827725    4960 scope.go:122] "RemoveContainer" containerID="67fccd74e6c51ed68503a59c82adcb319ba254d10047130460a1e7edca3ed154"
Oct 30 03:43:02.828102 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:43:02.828077    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67fccd74e6c51ed68503a59c82adcb319ba254d10047130460a1e7edca3ed154\": container with ID starting with 67fccd74e6c51ed68503a59c82adcb319ba254d10047130460a1e7edca3ed154 not found: ID does not exist" containerID="67fccd74e6c51ed68503a59c82adcb319ba254d10047130460a1e7edca3ed154"
Oct 30 03:43:02.828229 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.828113    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67fccd74e6c51ed68503a59c82adcb319ba254d10047130460a1e7edca3ed154"} err="failed to get container status \"67fccd74e6c51ed68503a59c82adcb319ba254d10047130460a1e7edca3ed154\": rpc error: code = NotFound desc = could not find container \"67fccd74e6c51ed68503a59c82adcb319ba254d10047130460a1e7edca3ed154\": container with ID starting with 67fccd74e6c51ed68503a59c82adcb319ba254d10047130460a1e7edca3ed154 not found: ID does not exist"
Oct 30 03:43:02.828229 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.828136    4960 scope.go:122] "RemoveContainer" containerID="a8cdaa8570554a3e7988451326a7e52f33c1c0394c0cf0edcae2ba5c9d252f23"
Oct 30 03:43:02.838336 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.838318    4960 scope.go:122] "RemoveContainer" containerID="a8cdaa8570554a3e7988451326a7e52f33c1c0394c0cf0edcae2ba5c9d252f23"
Oct 30 03:43:02.838585 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:43:02.838569    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8cdaa8570554a3e7988451326a7e52f33c1c0394c0cf0edcae2ba5c9d252f23\": container with ID starting with a8cdaa8570554a3e7988451326a7e52f33c1c0394c0cf0edcae2ba5c9d252f23 not found: ID does not exist" containerID="a8cdaa8570554a3e7988451326a7e52f33c1c0394c0cf0edcae2ba5c9d252f23"
Oct 30 03:43:02.838646 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.838592    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8cdaa8570554a3e7988451326a7e52f33c1c0394c0cf0edcae2ba5c9d252f23"} err="failed to get container status \"a8cdaa8570554a3e7988451326a7e52f33c1c0394c0cf0edcae2ba5c9d252f23\": rpc error: code = NotFound desc = could not find container \"a8cdaa8570554a3e7988451326a7e52f33c1c0394c0cf0edcae2ba5c9d252f23\": container with ID starting with a8cdaa8570554a3e7988451326a7e52f33c1c0394c0cf0edcae2ba5c9d252f23 not found: ID does not exist"
Oct 30 03:43:02.871285 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.871265    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-4pzml"]
Oct 30 03:43:02.875007 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.874984    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-4pzml"]
Oct 30 03:43:02.911343 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.911320    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:02.917239 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.917221    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:02.918139 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:02.918121    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:03.503062 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:03.503029    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29798940-9dfd-4438-b3cf-c0992468ba16" path="/var/lib/kubelet/pods/29798940-9dfd-4438-b3cf-c0992468ba16/volumes"
Oct 30 03:43:03.503475 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:03.503461    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1" path="/var/lib/kubelet/pods/45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1/volumes"
Oct 30 03:43:06.030119 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:06.030085    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:23.953245 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:43:23.953205    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:43:29.572827 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.572795    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"]
Oct 30 03:43:29.573234 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.573086    4960 cpu_manager.go:414] "containerMap: removing container" podUID="45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1" containerName="korrel8r"
Oct 30 03:43:29.573234 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.573097    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1" containerName="korrel8r"
Oct 30 03:43:29.573234 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.573130    4960 cpu_manager.go:414] "containerMap: removing container" podUID="29798940-9dfd-4438-b3cf-c0992468ba16" containerName="otc-container"
Oct 30 03:43:29.573234 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.573135    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29798940-9dfd-4438-b3cf-c0992468ba16" containerName="otc-container"
Oct 30 03:43:29.573234 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.573196    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="29798940-9dfd-4438-b3cf-c0992468ba16" containerName="otc-container"
Oct 30 03:43:29.573234 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.573206    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f61d6b-fd6d-43c7-ab4f-a36fc5fe2ff1" containerName="korrel8r"
Oct 30 03:43:29.576218 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.576201    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"
Oct 30 03:43:29.601706 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.601682    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:43:29.601824 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.601748    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:43:29.613614 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.613592    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-mm9g5\""
Oct 30 03:43:29.613715 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.613608    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\""
Oct 30 03:43:29.614860 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.614838    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"]
Oct 30 03:43:29.615104 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.615082    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:29.615575 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.615561    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\""
Oct 30 03:43:29.675965 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.675942    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-tls\") pod \"korrel8r-5d45699dc4-2q44x\" (UID: \"86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"
Oct 30 03:43:29.676082 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.675985    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkm77\" (UniqueName: \"kubernetes.io/projected/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-kube-api-access-jkm77\") pod \"korrel8r-5d45699dc4-2q44x\" (UID: \"86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"
Oct 30 03:43:29.676150 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.676130    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-config\") pod \"korrel8r-5d45699dc4-2q44x\" (UID: \"86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"
Oct 30 03:43:29.761377 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.761351    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:29.776698 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.776675    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-config\") pod \"korrel8r-5d45699dc4-2q44x\" (UID: \"86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"
Oct 30 03:43:29.776794 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.776708    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-tls\") pod \"korrel8r-5d45699dc4-2q44x\" (UID: \"86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"
Oct 30 03:43:29.776794 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.776739    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkm77\" (UniqueName: \"kubernetes.io/projected/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-kube-api-access-jkm77\") pod \"korrel8r-5d45699dc4-2q44x\" (UID: \"86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"
Oct 30 03:43:29.777471 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.777452    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-config\") pod \"korrel8r-5d45699dc4-2q44x\" (UID: \"86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"
Oct 30 03:43:29.779265 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.779243    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-tls\") pod \"korrel8r-5d45699dc4-2q44x\" (UID: \"86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"
Oct 30 03:43:29.798558 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.798534    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkm77\" (UniqueName: \"kubernetes.io/projected/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-kube-api-access-jkm77\") pod \"korrel8r-5d45699dc4-2q44x\" (UID: \"86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"
Oct 30 03:43:29.888609 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:29.888557    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"
Oct 30 03:43:30.185279 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.185244    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"]
Oct 30 03:43:30.194927 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:43:30.194891    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d3e4b9_ee0a_4428_8cd8_5c982ed7d2e0.slice/crio-721bdbc98ddbbf9cdd3226e389ddfa7c1fbfbe681eedad223b058a861c1ac3f3 WatchSource:0}: Error finding container 721bdbc98ddbbf9cdd3226e389ddfa7c1fbfbe681eedad223b058a861c1ac3f3: Status 404 returned error can't find the container with id 721bdbc98ddbbf9cdd3226e389ddfa7c1fbfbe681eedad223b058a861c1ac3f3
Oct 30 03:43:30.226947 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.226917    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:30.231770 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.231749    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:30.245364 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.245338    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:30.255295 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.255272    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:30.287687 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.287662    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:30.287812 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.287715    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:30.287812 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.287740    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:30.287812 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.287777    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxxjz\" (UniqueName: \"kubernetes.io/projected/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-kube-api-access-zxxjz\") pod \"data-science-collector-collector-1\" (UID: \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:30.316043 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.316016    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:43:30.316158 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.316057    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-7g6tv\""
Oct 30 03:43:30.316472 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.316454    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\""
Oct 30 03:43:30.316539 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.316479    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:43:30.379578 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.379555    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:30.388676 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.388651    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:30.388778 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.388683    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:30.388778 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.388716    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxxjz\" (UniqueName: \"kubernetes.io/projected/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-kube-api-access-zxxjz\") pod \"data-science-collector-collector-1\" (UID: \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:30.388778 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.388747    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:30.389344 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.389322    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:30.391457 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.391432    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:30.391559 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.391543    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:30.413226 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.413205    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxxjz\" (UniqueName: \"kubernetes.io/projected/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-kube-api-access-zxxjz\") pod \"data-science-collector-collector-1\" (UID: \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:30.555007 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.554944    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:30.610301 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.608901    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:30.850876 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.850850    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:30.853644 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:43:30.853616    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bfd3adc_3e35_46ea_8bc9_d10830e4fa42.slice/crio-cb6c6d0bf5448f66d6e3ae7142ad7f8bb80c7db9b6dbd24614d3f7cfd27a2d01 WatchSource:0}: Error finding container cb6c6d0bf5448f66d6e3ae7142ad7f8bb80c7db9b6dbd24614d3f7cfd27a2d01: Status 404 returned error can't find the container with id cb6c6d0bf5448f66d6e3ae7142ad7f8bb80c7db9b6dbd24614d3f7cfd27a2d01
Oct 30 03:43:30.856258 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.856231    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:43:30.856331 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.856286    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:43:30.946859 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.946267    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42","Type":"ContainerStarted","Data":"53a43f223ecac41484249feaad336dfd402879fa4347a4b9f5936e566d146d02"}
Oct 30 03:43:30.946859 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.946350    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42","Type":"ContainerStarted","Data":"cb6c6d0bf5448f66d6e3ae7142ad7f8bb80c7db9b6dbd24614d3f7cfd27a2d01"}
Oct 30 03:43:30.948009 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.947982    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x" event={"ID":"86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0","Type":"ContainerStarted","Data":"a0180660c9d824de668a3c9d369c3e013a982785649c3833e6580bf69638b0bc"}
Oct 30 03:43:30.948133 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.948035    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x" event={"ID":"86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0","Type":"ContainerStarted","Data":"721bdbc98ddbbf9cdd3226e389ddfa7c1fbfbe681eedad223b058a861c1ac3f3"}
Oct 30 03:43:30.948133 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.948073    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"
Oct 30 03:43:30.991907 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:30.991843    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=0.991824639 podStartE2EDuration="991.824639ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:30.990542085 +0000 UTC m=+3691.639590506" watchObservedRunningTime="2026-10-30 03:43:30.991824639 +0000 UTC m=+3691.640873061"
Oct 30 03:43:31.034219 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:31.034130    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x" podStartSLOduration=2.034117143 podStartE2EDuration="2.034117143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:31.03389023 +0000 UTC m=+3691.682938652" watchObservedRunningTime="2026-10-30 03:43:31.034117143 +0000 UTC m=+3691.683165561"
Oct 30 03:43:33.049719 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.049677    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:33.186452 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.186415    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:33.307134 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.307052    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:33.431276 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.431245    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:33.568120 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.568088    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:33.595473 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.595447    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"]
Oct 30 03:43:33.599017 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.598999    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.629789 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.629760    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\""
Oct 30 03:43:33.630098 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.630080    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\""
Oct 30 03:43:33.630098 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.630082    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-brrkd\""
Oct 30 03:43:33.630268 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.630138    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\""
Oct 30 03:43:33.630268 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.630151    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\""
Oct 30 03:43:33.630268 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.630170    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\""
Oct 30 03:43:33.630419 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.630271    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\""
Oct 30 03:43:33.631498 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.631475    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"]
Oct 30 03:43:33.716232 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.716207    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.716394 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.716245    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-tenant\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.716490 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.716463    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-serving-certs\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.716613 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.716578    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-rbac\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.716688 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.716644    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.716688 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.716675    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-cabundle\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.716773 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.716702    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gh5k\" (UniqueName: \"kubernetes.io/projected/b41aae44-16cf-4292-a557-dd904dfd6875-kube-api-access-8gh5k\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.818093 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.818064    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.818269 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.818100    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-cabundle\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.818269 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.818124    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gh5k\" (UniqueName: \"kubernetes.io/projected/b41aae44-16cf-4292-a557-dd904dfd6875-kube-api-access-8gh5k\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.818269 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.818168    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.818269 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.818220    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-tenant\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.818500 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.818289    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-serving-certs\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.818500 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.818322    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-rbac\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.818990 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.818967    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.819102 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.818967    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-rbac\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.819102 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.819066    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-cabundle\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.820906 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.820883    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.820906 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.820893    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-tenant\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.821015 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.820924    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-serving-certs\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.839345 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.839322    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gh5k\" (UniqueName: \"kubernetes.io/projected/b41aae44-16cf-4292-a557-dd904dfd6875-kube-api-access-8gh5k\") pod \"tempo-data-science-tempostack-gateway-945766f69-qwtsd\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:33.914714 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:33.914691    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:34.231357 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:34.231331    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"]
Oct 30 03:43:34.233109 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:43:34.233071    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41aae44_16cf_4292_a557_dd904dfd6875.slice/crio-c32c2bb3cd98991e9de64c47a93fe880754cb257aee012e48efa56591cdab257 WatchSource:0}: Error finding container c32c2bb3cd98991e9de64c47a93fe880754cb257aee012e48efa56591cdab257: Status 404 returned error can't find the container with id c32c2bb3cd98991e9de64c47a93fe880754cb257aee012e48efa56591cdab257
Oct 30 03:43:34.235030 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:34.235013    4960 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:43:34.970686 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:34.970650    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd" event={"ID":"b41aae44-16cf-4292-a557-dd904dfd6875","Type":"ContainerStarted","Data":"c32c2bb3cd98991e9de64c47a93fe880754cb257aee012e48efa56591cdab257"}
Oct 30 03:43:35.328232 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:35.328193    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:35.445803 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:35.445775    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:36.206565 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:36.206536    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:36.982693 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:36.982655    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd" event={"ID":"b41aae44-16cf-4292-a557-dd904dfd6875","Type":"ContainerStarted","Data":"963c87af68da21581757d93b5ab1edf84f64057e4a0fdcc51bc37c2cc770de28"}
Oct 30 03:43:37.660291 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:37.660247    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"]
Oct 30 03:43:38.993866 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:38.993818    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd" event={"ID":"b41aae44-16cf-4292-a557-dd904dfd6875","Type":"ContainerStarted","Data":"9c0154de9cb4c4c8a5dad6350755b351a497b6036d48fd2320add77e2374d88b"}
Oct 30 03:43:38.994418 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:38.993870    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd" podUID="b41aae44-16cf-4292-a557-dd904dfd6875" containerName="tempo-gateway" containerID="cri-o://963c87af68da21581757d93b5ab1edf84f64057e4a0fdcc51bc37c2cc770de28" gracePeriod=30
Oct 30 03:43:38.994418 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:38.993935    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd" podUID="b41aae44-16cf-4292-a557-dd904dfd6875" containerName="tempo-gateway-opa" containerID="cri-o://9c0154de9cb4c4c8a5dad6350755b351a497b6036d48fd2320add77e2374d88b" gracePeriod=30
Oct 30 03:43:38.994418 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:38.994027    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:38.994418 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:38.994207    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:38.995904 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:38.995866    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd" podUID="b41aae44-16cf-4292-a557-dd904dfd6875" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.70:8083/ready\": dial tcp 10.132.0.70:8083: connect: connection refused"
Oct 30 03:43:38.999824 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:38.999791    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd" podUID="b41aae44-16cf-4292-a557-dd904dfd6875" containerName="tempo-gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:43:39.059155 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.059118    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd" podStartSLOduration=2.529235499 podStartE2EDuration="6.059105927s" totalImagesPullingTime="3.529870428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:39.058956741 +0000 UTC m=+3699.708005161" watchObservedRunningTime="2026-10-30 03:43:39.059105927 +0000 UTC m=+3699.708154347"
Oct 30 03:43:39.252645 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.252590    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:39.361748 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.361718    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-rbac\" (UniqueName: \"kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-rbac\") pod \"b41aae44-16cf-4292-a557-dd904dfd6875\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") "
Oct 30 03:43:39.361893 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.361766    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-tempo-data-science-tempostack-gateway-mtls\") pod \"b41aae44-16cf-4292-a557-dd904dfd6875\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") "
Oct 30 03:43:39.361893 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.361800    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-tempo-data-science-tempostack-ca-bundle\") pod \"b41aae44-16cf-4292-a557-dd904dfd6875\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") "
Oct 30 03:43:39.362002 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.361955    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-tenant\" (UniqueName: \"kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-tenant\") pod \"b41aae44-16cf-4292-a557-dd904dfd6875\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") "
Oct 30 03:43:39.362055 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.362006    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-cabundle\" (UniqueName: \"kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-cabundle\") pod \"b41aae44-16cf-4292-a557-dd904dfd6875\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") "
Oct 30 03:43:39.362055 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.362027    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b41aae44-16cf-4292-a557-dd904dfd6875-kube-api-access-8gh5k\" (UniqueName: \"kubernetes.io/projected/b41aae44-16cf-4292-a557-dd904dfd6875-kube-api-access-8gh5k\") pod \"b41aae44-16cf-4292-a557-dd904dfd6875\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") "
Oct 30 03:43:39.362167 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.362056    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-serving-certs\" (UniqueName: \"kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-serving-certs\") pod \"b41aae44-16cf-4292-a557-dd904dfd6875\" (UID: \"b41aae44-16cf-4292-a557-dd904dfd6875\") "
Oct 30 03:43:39.362167 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.362133    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-rbac" pod "b41aae44-16cf-4292-a557-dd904dfd6875" (UID: "b41aae44-16cf-4292-a557-dd904dfd6875"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:39.362306 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.362203    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-tempo-data-science-tempostack-ca-bundle" pod "b41aae44-16cf-4292-a557-dd904dfd6875" (UID: "b41aae44-16cf-4292-a557-dd904dfd6875"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:39.362454 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.362423    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-cabundle" pod "b41aae44-16cf-4292-a557-dd904dfd6875" (UID: "b41aae44-16cf-4292-a557-dd904dfd6875"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:39.362596 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.362457    4960 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-rbac\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:43:39.362596 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.362482    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:43:39.364198 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.364157    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-tempo-data-science-tempostack-gateway-mtls" pod "b41aae44-16cf-4292-a557-dd904dfd6875" (UID: "b41aae44-16cf-4292-a557-dd904dfd6875"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:39.364303 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.364260    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41aae44-16cf-4292-a557-dd904dfd6875-kube-api-access-8gh5k" pod "b41aae44-16cf-4292-a557-dd904dfd6875" (UID: "b41aae44-16cf-4292-a557-dd904dfd6875"). InnerVolumeSpecName "kube-api-access-8gh5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:39.364372 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.364311    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-serving-certs" pod "b41aae44-16cf-4292-a557-dd904dfd6875" (UID: "b41aae44-16cf-4292-a557-dd904dfd6875"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:39.368563 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.368539    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-tenant" pod "b41aae44-16cf-4292-a557-dd904dfd6875" (UID: "b41aae44-16cf-4292-a557-dd904dfd6875"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:39.463493 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.463472    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:43:39.463590 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.463499    4960 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-tenant\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:43:39.463590 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.463513    4960 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/b41aae44-16cf-4292-a557-dd904dfd6875-cabundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:43:39.463590 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.463523    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gh5k\" (UniqueName: \"kubernetes.io/projected/b41aae44-16cf-4292-a557-dd904dfd6875-kube-api-access-8gh5k\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:43:39.463590 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:39.463532    4960 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/b41aae44-16cf-4292-a557-dd904dfd6875-serving-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:43:40.001107 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:40.001073    4960 generic.go:356] "Generic (PLEG): container finished" podID="b41aae44-16cf-4292-a557-dd904dfd6875" containerID="9c0154de9cb4c4c8a5dad6350755b351a497b6036d48fd2320add77e2374d88b" exitCode=2
Oct 30 03:43:40.001107 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:40.001101    4960 generic.go:356] "Generic (PLEG): container finished" podID="b41aae44-16cf-4292-a557-dd904dfd6875" containerID="963c87af68da21581757d93b5ab1edf84f64057e4a0fdcc51bc37c2cc770de28" exitCode=0
Oct 30 03:43:40.001605 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:40.001126    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd" event={"ID":"b41aae44-16cf-4292-a557-dd904dfd6875","Type":"ContainerDied","Data":"9c0154de9cb4c4c8a5dad6350755b351a497b6036d48fd2320add77e2374d88b"}
Oct 30 03:43:40.001605 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:40.001138    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"
Oct 30 03:43:40.001605 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:40.001196    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd" event={"ID":"b41aae44-16cf-4292-a557-dd904dfd6875","Type":"ContainerDied","Data":"963c87af68da21581757d93b5ab1edf84f64057e4a0fdcc51bc37c2cc770de28"}
Oct 30 03:43:40.001605 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:40.001219    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd" event={"ID":"b41aae44-16cf-4292-a557-dd904dfd6875","Type":"ContainerDied","Data":"c32c2bb3cd98991e9de64c47a93fe880754cb257aee012e48efa56591cdab257"}
Oct 30 03:43:40.001605 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:40.001247    4960 scope.go:122] "RemoveContainer" containerID="9c0154de9cb4c4c8a5dad6350755b351a497b6036d48fd2320add77e2374d88b"
Oct 30 03:43:40.010330 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:40.010310    4960 scope.go:122] "RemoveContainer" containerID="963c87af68da21581757d93b5ab1edf84f64057e4a0fdcc51bc37c2cc770de28"
Oct 30 03:43:40.018794 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:40.018775    4960 scope.go:122] "RemoveContainer" containerID="9c0154de9cb4c4c8a5dad6350755b351a497b6036d48fd2320add77e2374d88b"
Oct 30 03:43:40.019097 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:43:40.019076    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c0154de9cb4c4c8a5dad6350755b351a497b6036d48fd2320add77e2374d88b\": container with ID starting with 9c0154de9cb4c4c8a5dad6350755b351a497b6036d48fd2320add77e2374d88b not found: ID does not exist" containerID="9c0154de9cb4c4c8a5dad6350755b351a497b6036d48fd2320add77e2374d88b"
Oct 30 03:43:40.019154 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:40.019106    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c0154de9cb4c4c8a5dad6350755b351a497b6036d48fd2320add77e2374d88b"} err="failed to get container status \"9c0154de9cb4c4c8a5dad6350755b351a497b6036d48fd2320add77e2374d88b\": rpc error: code = NotFound desc = could not find container \"9c0154de9cb4c4c8a5dad6350755b351a497b6036d48fd2320add77e2374d88b\": container with ID starting with 9c0154de9cb4c4c8a5dad6350755b351a497b6036d48fd2320add77e2374d88b not found: ID does not exist"
Oct 30 03:43:40.019154 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:40.019122    4960 scope.go:122] "RemoveContainer" containerID="963c87af68da21581757d93b5ab1edf84f64057e4a0fdcc51bc37c2cc770de28"
Oct 30 03:43:40.019367 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:43:40.019351    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963c87af68da21581757d93b5ab1edf84f64057e4a0fdcc51bc37c2cc770de28\": container with ID starting with 963c87af68da21581757d93b5ab1edf84f64057e4a0fdcc51bc37c2cc770de28 not found: ID does not exist" containerID="963c87af68da21581757d93b5ab1edf84f64057e4a0fdcc51bc37c2cc770de28"
Oct 30 03:43:40.019404 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:40.019370    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963c87af68da21581757d93b5ab1edf84f64057e4a0fdcc51bc37c2cc770de28"} err="failed to get container status \"963c87af68da21581757d93b5ab1edf84f64057e4a0fdcc51bc37c2cc770de28\": rpc error: code = NotFound desc = could not find container \"963c87af68da21581757d93b5ab1edf84f64057e4a0fdcc51bc37c2cc770de28\": container with ID starting with 963c87af68da21581757d93b5ab1edf84f64057e4a0fdcc51bc37c2cc770de28 not found: ID does not exist"
Oct 30 03:43:40.019404 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:40.019383    4960 scope.go:122] "RemoveContainer" containerID="9c0154de9cb4c4c8a5dad6350755b351a497b6036d48fd2320add77e2374d88b"
Oct 30 03:43:40.019623 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:40.019602    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c0154de9cb4c4c8a5dad6350755b351a497b6036d48fd2320add77e2374d88b"} err="failed to get container status \"9c0154de9cb4c4c8a5dad6350755b351a497b6036d48fd2320add77e2374d88b\": rpc error: code = NotFound desc = could not find container \"9c0154de9cb4c4c8a5dad6350755b351a497b6036d48fd2320add77e2374d88b\": container with ID starting with 9c0154de9cb4c4c8a5dad6350755b351a497b6036d48fd2320add77e2374d88b not found: ID does not exist"
Oct 30 03:43:40.019681 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:40.019624    4960 scope.go:122] "RemoveContainer" containerID="963c87af68da21581757d93b5ab1edf84f64057e4a0fdcc51bc37c2cc770de28"
Oct 30 03:43:40.019831 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:40.019810    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963c87af68da21581757d93b5ab1edf84f64057e4a0fdcc51bc37c2cc770de28"} err="failed to get container status \"963c87af68da21581757d93b5ab1edf84f64057e4a0fdcc51bc37c2cc770de28\": rpc error: code = NotFound desc = could not find container \"963c87af68da21581757d93b5ab1edf84f64057e4a0fdcc51bc37c2cc770de28\": container with ID starting with 963c87af68da21581757d93b5ab1edf84f64057e4a0fdcc51bc37c2cc770de28 not found: ID does not exist"
Oct 30 03:43:40.071361 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:40.071341    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"]
Oct 30 03:43:40.085022 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:40.085002    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-945766f69-qwtsd"]
Oct 30 03:43:41.508198 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:41.508152    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b41aae44-16cf-4292-a557-dd904dfd6875" path="/var/lib/kubelet/pods/b41aae44-16cf-4292-a557-dd904dfd6875/volumes"
Oct 30 03:43:41.955302 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:41.955268    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"
Oct 30 03:43:42.179505 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:42.179452    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:52.099798 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:52.099761    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:53.953936 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:43:53.953894    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:43:56.625395 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:43:56.625361    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:23.954988 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:44:23.954909    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:44:38.420131 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:38.420093    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:38.567168 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:38.567133    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:38.964693 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:38.964665    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:38.964900 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:38.964868    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="0bfd3adc-3e35-46ea-8bc9-d10830e4fa42" containerName="otc-container" containerID="cri-o://53a43f223ecac41484249feaad336dfd402879fa4347a4b9f5936e566d146d02" gracePeriod=30
Oct 30 03:44:39.123166 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.123125    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:39.217415 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.217358    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:39.242764 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.242717    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:39.266603 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.266471    4960 generic.go:356] "Generic (PLEG): container finished" podID="0bfd3adc-3e35-46ea-8bc9-d10830e4fa42" containerID="53a43f223ecac41484249feaad336dfd402879fa4347a4b9f5936e566d146d02" exitCode=0
Oct 30 03:44:39.266603 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.266569    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:39.267341 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.266580    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42","Type":"ContainerDied","Data":"53a43f223ecac41484249feaad336dfd402879fa4347a4b9f5936e566d146d02"}
Oct 30 03:44:39.267341 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.266693    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42","Type":"ContainerDied","Data":"cb6c6d0bf5448f66d6e3ae7142ad7f8bb80c7db9b6dbd24614d3f7cfd27a2d01"}
Oct 30 03:44:39.267341 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.266739    4960 scope.go:122] "RemoveContainer" containerID="53a43f223ecac41484249feaad336dfd402879fa4347a4b9f5936e566d146d02"
Oct 30 03:44:39.278612 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.278579    4960 scope.go:122] "RemoveContainer" containerID="53a43f223ecac41484249feaad336dfd402879fa4347a4b9f5936e566d146d02"
Oct 30 03:44:39.278930 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:44:39.278910    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53a43f223ecac41484249feaad336dfd402879fa4347a4b9f5936e566d146d02\": container with ID starting with 53a43f223ecac41484249feaad336dfd402879fa4347a4b9f5936e566d146d02 not found: ID does not exist" containerID="53a43f223ecac41484249feaad336dfd402879fa4347a4b9f5936e566d146d02"
Oct 30 03:44:39.279014 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.278939    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a43f223ecac41484249feaad336dfd402879fa4347a4b9f5936e566d146d02"} err="failed to get container status \"53a43f223ecac41484249feaad336dfd402879fa4347a4b9f5936e566d146d02\": rpc error: code = NotFound desc = could not find container \"53a43f223ecac41484249feaad336dfd402879fa4347a4b9f5936e566d146d02\": container with ID starting with 53a43f223ecac41484249feaad336dfd402879fa4347a4b9f5936e566d146d02 not found: ID does not exist"
Oct 30 03:44:39.306523 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.306497    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-otlp-tls-certs\") pod \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\" (UID: \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\") "
Oct 30 03:44:39.306636 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.306548    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-kube-api-access-zxxjz\" (UniqueName: \"kubernetes.io/projected/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-kube-api-access-zxxjz\") pod \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\" (UID: \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\") "
Oct 30 03:44:39.306636 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.306609    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-monitoring-tls-certs\") pod \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\" (UID: \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\") "
Oct 30 03:44:39.306723 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.306658    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-otc-internal\" (UniqueName: \"kubernetes.io/configmap/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-otc-internal\") pod \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\" (UID: \"0bfd3adc-3e35-46ea-8bc9-d10830e4fa42\") "
Oct 30 03:44:39.307011 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.306989    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-otc-internal" pod "0bfd3adc-3e35-46ea-8bc9-d10830e4fa42" (UID: "0bfd3adc-3e35-46ea-8bc9-d10830e4fa42"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:39.308828 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.308781    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-kube-api-access-zxxjz" pod "0bfd3adc-3e35-46ea-8bc9-d10830e4fa42" (UID: "0bfd3adc-3e35-46ea-8bc9-d10830e4fa42"). InnerVolumeSpecName "kube-api-access-zxxjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:39.308828 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.308811    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-otlp-tls-certs" pod "0bfd3adc-3e35-46ea-8bc9-d10830e4fa42" (UID: "0bfd3adc-3e35-46ea-8bc9-d10830e4fa42"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:39.308942 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.308851    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-monitoring-tls-certs" pod "0bfd3adc-3e35-46ea-8bc9-d10830e4fa42" (UID: "0bfd3adc-3e35-46ea-8bc9-d10830e4fa42"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:39.407785 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.407762    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:44:39.407785 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.407784    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:44:39.407916 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.407793    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:44:39.407916 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.407802    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxxjz\" (UniqueName: \"kubernetes.io/projected/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42-kube-api-access-zxxjz\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:44:39.618293 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.618267    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:39.639962 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.639934    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:39.689001 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.688976    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:39.689328 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.689313    4960 cpu_manager.go:414] "containerMap: removing container" podUID="b41aae44-16cf-4292-a557-dd904dfd6875" containerName="tempo-gateway"
Oct 30 03:44:39.689391 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.689332    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41aae44-16cf-4292-a557-dd904dfd6875" containerName="tempo-gateway"
Oct 30 03:44:39.689391 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.689345    4960 cpu_manager.go:414] "containerMap: removing container" podUID="b41aae44-16cf-4292-a557-dd904dfd6875" containerName="tempo-gateway-opa"
Oct 30 03:44:39.689391 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.689353    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41aae44-16cf-4292-a557-dd904dfd6875" containerName="tempo-gateway-opa"
Oct 30 03:44:39.689391 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.689373    4960 cpu_manager.go:414] "containerMap: removing container" podUID="0bfd3adc-3e35-46ea-8bc9-d10830e4fa42" containerName="otc-container"
Oct 30 03:44:39.689391 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.689378    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bfd3adc-3e35-46ea-8bc9-d10830e4fa42" containerName="otc-container"
Oct 30 03:44:39.689547 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.689439    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41aae44-16cf-4292-a557-dd904dfd6875" containerName="tempo-gateway"
Oct 30 03:44:39.689547 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.689447    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41aae44-16cf-4292-a557-dd904dfd6875" containerName="tempo-gateway-opa"
Oct 30 03:44:39.689547 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.689457    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bfd3adc-3e35-46ea-8bc9-d10830e4fa42" containerName="otc-container"
Oct 30 03:44:39.692168 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.692148    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:39.710030 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.710005    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:39.710130 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.710035    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lmxb\" (UniqueName: \"kubernetes.io/projected/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-kube-api-access-8lmxb\") pod \"data-science-collector-collector-1\" (UID: \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:39.710130 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.710056    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:39.710130 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.710076    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:39.710747 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.710731    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:44:39.710934 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.710915    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\""
Oct 30 03:44:39.711022 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.711011    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-7g6tv\""
Oct 30 03:44:39.711055 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.711022    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:44:39.725679 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.725655    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:39.811001 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.810945    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:39.811001 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.810982    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lmxb\" (UniqueName: \"kubernetes.io/projected/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-kube-api-access-8lmxb\") pod \"data-science-collector-collector-1\" (UID: \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:39.811193 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.811013    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:39.811193 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.811044    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:39.811759 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.811705    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:39.813534 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.813516    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:39.813655 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.813532    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:39.832801 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:39.832775    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lmxb\" (UniqueName: \"kubernetes.io/projected/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-kube-api-access-8lmxb\") pod \"data-science-collector-collector-1\" (UID: \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:40.002880 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:40.002851    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:40.290292 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:40.290269    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:40.291875 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:44:40.291844    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08c2c87_8b51_4e85_b86c_bcfc4f605e7b.slice/crio-32ad4ef07a539a325b992d96c480c9efd0a4a5ee1e87bf1385bc80110ac9d7b2 WatchSource:0}: Error finding container 32ad4ef07a539a325b992d96c480c9efd0a4a5ee1e87bf1385bc80110ac9d7b2: Status 404 returned error can't find the container with id 32ad4ef07a539a325b992d96c480c9efd0a4a5ee1e87bf1385bc80110ac9d7b2
Oct 30 03:44:40.294034 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:40.293998    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:44:40.294117 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:40.294072    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:44:41.277363 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:41.277329    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b","Type":"ContainerStarted","Data":"6d368e473f8d1a4866a1232f73f327db9c41701ced3d4f101cbd33f41c369468"}
Oct 30 03:44:41.277738 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:41.277377    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b","Type":"ContainerStarted","Data":"32ad4ef07a539a325b992d96c480c9efd0a4a5ee1e87bf1385bc80110ac9d7b2"}
Oct 30 03:44:41.331016 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:41.330964    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.330950757 podStartE2EDuration="2.330950757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:41.329642944 +0000 UTC m=+3761.978691365" watchObservedRunningTime="2026-10-30 03:44:41.330950757 +0000 UTC m=+3761.979999178"
Oct 30 03:44:41.504084 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:41.504050    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bfd3adc-3e35-46ea-8bc9-d10830e4fa42" path="/var/lib/kubelet/pods/0bfd3adc-3e35-46ea-8bc9-d10830e4fa42/volumes"
Oct 30 03:44:42.132679 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:42.132645    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"]
Oct 30 03:44:42.132862 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:42.132842    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x" podUID="86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0" containerName="korrel8r" containerID="cri-o://a0180660c9d824de668a3c9d369c3e013a982785649c3833e6580bf69638b0bc" gracePeriod=30
Oct 30 03:44:42.283387 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:42.283358    4960 generic.go:356] "Generic (PLEG): container finished" podID="86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0" containerID="a0180660c9d824de668a3c9d369c3e013a982785649c3833e6580bf69638b0bc" exitCode=0
Oct 30 03:44:42.283781 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:42.283416    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x" event={"ID":"86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0","Type":"ContainerDied","Data":"a0180660c9d824de668a3c9d369c3e013a982785649c3833e6580bf69638b0bc"}
Oct 30 03:44:42.380136 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:42.380109    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"
Oct 30 03:44:42.431990 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:42.431914    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-kube-api-access-jkm77\" (UniqueName: \"kubernetes.io/projected/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-kube-api-access-jkm77\") pod \"86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0\" (UID: \"86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0\") "
Oct 30 03:44:42.431990 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:42.431969    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-tls\" (UniqueName: \"kubernetes.io/secret/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-tls\") pod \"86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0\" (UID: \"86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0\") "
Oct 30 03:44:42.432211 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:42.432001    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-config\" (UniqueName: \"kubernetes.io/configmap/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-config\") pod \"86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0\" (UID: \"86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0\") "
Oct 30 03:44:42.432454 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:42.432430    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-config" pod "86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0" (UID: "86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:42.434424 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:42.434399    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-tls" pod "86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0" (UID: "86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:42.434610 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:42.434595    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-kube-api-access-jkm77" pod "86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0" (UID: "86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0"). InnerVolumeSpecName "kube-api-access-jkm77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:42.533380 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:42.533352    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:44:42.533380 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:42.533380    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:44:42.533558 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:42.533393    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkm77\" (UniqueName: \"kubernetes.io/projected/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0-kube-api-access-jkm77\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:44:42.554806 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:42.554777    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:42.603830 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:42.603803    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:42.808573 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:42.808498    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:42.935288 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:42.935252    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:43.181773 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.181744    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:43.290252 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.290223    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"
Oct 30 03:44:43.290252 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.290224    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x" event={"ID":"86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0","Type":"ContainerDied","Data":"721bdbc98ddbbf9cdd3226e389ddfa7c1fbfbe681eedad223b058a861c1ac3f3"}
Oct 30 03:44:43.290780 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.290310    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="f08c2c87-8b51-4e85-b86c-bcfc4f605e7b" containerName="otc-container" containerID="cri-o://6d368e473f8d1a4866a1232f73f327db9c41701ced3d4f101cbd33f41c369468" gracePeriod=30
Oct 30 03:44:43.290780 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.290320    4960 scope.go:122] "RemoveContainer" containerID="a0180660c9d824de668a3c9d369c3e013a982785649c3833e6580bf69638b0bc"
Oct 30 03:44:43.334083 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.334053    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:43.412412 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.412384    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"]
Oct 30 03:44:43.417841 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.417820    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-2q44x"]
Oct 30 03:44:43.502989 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.502960    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0" path="/var/lib/kubelet/pods/86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0/volumes"
Oct 30 03:44:43.543380 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.543359    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:43.642307 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.642281    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-otlp-tls-certs\") pod \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\" (UID: \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\") "
Oct 30 03:44:43.642453 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.642313    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-otc-internal\" (UniqueName: \"kubernetes.io/configmap/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-otc-internal\") pod \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\" (UID: \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\") "
Oct 30 03:44:43.642453 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.642345    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-monitoring-tls-certs\") pod \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\" (UID: \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\") "
Oct 30 03:44:43.642573 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.642482    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-kube-api-access-8lmxb\" (UniqueName: \"kubernetes.io/projected/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-kube-api-access-8lmxb\") pod \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\" (UID: \"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b\") "
Oct 30 03:44:43.642676 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.642657    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-otc-internal" pod "f08c2c87-8b51-4e85-b86c-bcfc4f605e7b" (UID: "f08c2c87-8b51-4e85-b86c-bcfc4f605e7b"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:43.642742 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.642719    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:44:43.644579 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.644553    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-otlp-tls-certs" pod "f08c2c87-8b51-4e85-b86c-bcfc4f605e7b" (UID: "f08c2c87-8b51-4e85-b86c-bcfc4f605e7b"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:43.644708 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.644643    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-kube-api-access-8lmxb" pod "f08c2c87-8b51-4e85-b86c-bcfc4f605e7b" (UID: "f08c2c87-8b51-4e85-b86c-bcfc4f605e7b"). InnerVolumeSpecName "kube-api-access-8lmxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:43.644708 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.644653    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-monitoring-tls-certs" pod "f08c2c87-8b51-4e85-b86c-bcfc4f605e7b" (UID: "f08c2c87-8b51-4e85-b86c-bcfc4f605e7b"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:43.744053 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.744002    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:44:43.744053 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.744022    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:44:43.744053 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:43.744033    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lmxb\" (UniqueName: \"kubernetes.io/projected/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b-kube-api-access-8lmxb\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.296489 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:44.296453    4960 generic.go:356] "Generic (PLEG): container finished" podID="f08c2c87-8b51-4e85-b86c-bcfc4f605e7b" containerID="6d368e473f8d1a4866a1232f73f327db9c41701ced3d4f101cbd33f41c369468" exitCode=0
Oct 30 03:44:44.296970 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:44.296506    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:44.296970 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:44.296533    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b","Type":"ContainerDied","Data":"6d368e473f8d1a4866a1232f73f327db9c41701ced3d4f101cbd33f41c369468"}
Oct 30 03:44:44.296970 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:44.296582    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f08c2c87-8b51-4e85-b86c-bcfc4f605e7b","Type":"ContainerDied","Data":"32ad4ef07a539a325b992d96c480c9efd0a4a5ee1e87bf1385bc80110ac9d7b2"}
Oct 30 03:44:44.296970 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:44.296604    4960 scope.go:122] "RemoveContainer" containerID="6d368e473f8d1a4866a1232f73f327db9c41701ced3d4f101cbd33f41c369468"
Oct 30 03:44:44.305886 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:44.305870    4960 scope.go:122] "RemoveContainer" containerID="6d368e473f8d1a4866a1232f73f327db9c41701ced3d4f101cbd33f41c369468"
Oct 30 03:44:44.306138 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:44:44.306120    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d368e473f8d1a4866a1232f73f327db9c41701ced3d4f101cbd33f41c369468\": container with ID starting with 6d368e473f8d1a4866a1232f73f327db9c41701ced3d4f101cbd33f41c369468 not found: ID does not exist" containerID="6d368e473f8d1a4866a1232f73f327db9c41701ced3d4f101cbd33f41c369468"
Oct 30 03:44:44.306228 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:44.306151    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d368e473f8d1a4866a1232f73f327db9c41701ced3d4f101cbd33f41c369468"} err="failed to get container status \"6d368e473f8d1a4866a1232f73f327db9c41701ced3d4f101cbd33f41c369468\": rpc error: code = NotFound desc = could not find container \"6d368e473f8d1a4866a1232f73f327db9c41701ced3d4f101cbd33f41c369468\": container with ID starting with 6d368e473f8d1a4866a1232f73f327db9c41701ced3d4f101cbd33f41c369468 not found: ID does not exist"
Oct 30 03:44:44.352910 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:44.352887    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:44.366572 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:44.366552    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:45.502267 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:45.502234    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f08c2c87-8b51-4e85-b86c-bcfc4f605e7b" path="/var/lib/kubelet/pods/f08c2c87-8b51-4e85-b86c-bcfc4f605e7b/volumes"
Oct 30 03:44:53.956093 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:44:53.956049    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:44:56.738525 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:44:56.738492    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:09.904993 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:09.904946    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:12.282635 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:12.282602    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:23.568925 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.568895    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:23.611433 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.611401    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"]
Oct 30 03:45:23.611748 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.611733    4960 cpu_manager.go:414] "containerMap: removing container" podUID="f08c2c87-8b51-4e85-b86c-bcfc4f605e7b" containerName="otc-container"
Oct 30 03:45:23.611792 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.611752    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f08c2c87-8b51-4e85-b86c-bcfc4f605e7b" containerName="otc-container"
Oct 30 03:45:23.611792 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.611770    4960 cpu_manager.go:414] "containerMap: removing container" podUID="86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0" containerName="korrel8r"
Oct 30 03:45:23.611792 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.611775    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0" containerName="korrel8r"
Oct 30 03:45:23.611888 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.611843    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d3e4b9-ee0a-4428-8cd8-5c982ed7d2e0" containerName="korrel8r"
Oct 30 03:45:23.611888 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.611856    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="f08c2c87-8b51-4e85-b86c-bcfc4f605e7b" containerName="otc-container"
Oct 30 03:45:23.614790 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.614771    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"
Oct 30 03:45:23.632066 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.632043    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:45:23.645728 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.645702    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\""
Oct 30 03:45:23.645728 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.645717    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-n72r4\""
Oct 30 03:45:23.645867 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.645773    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:45:23.645903 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.645875    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\""
Oct 30 03:45:23.646903 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.646884    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"]
Oct 30 03:45:23.722111 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.722089    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:23.727331 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.727289    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5d494afe-0126-4460-9f0a-614f6eb4f0f1-config\") pod \"korrel8r-5d45699dc4-4pv6g\" (UID: \"5d494afe-0126-4460-9f0a-614f6eb4f0f1\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"
Oct 30 03:45:23.727432 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.727343    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5d494afe-0126-4460-9f0a-614f6eb4f0f1-tls\") pod \"korrel8r-5d45699dc4-4pv6g\" (UID: \"5d494afe-0126-4460-9f0a-614f6eb4f0f1\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"
Oct 30 03:45:23.727474 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.727441    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gj46\" (UniqueName: \"kubernetes.io/projected/5d494afe-0126-4460-9f0a-614f6eb4f0f1-kube-api-access-8gj46\") pod \"korrel8r-5d45699dc4-4pv6g\" (UID: \"5d494afe-0126-4460-9f0a-614f6eb4f0f1\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"
Oct 30 03:45:23.828164 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.828116    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5d494afe-0126-4460-9f0a-614f6eb4f0f1-tls\") pod \"korrel8r-5d45699dc4-4pv6g\" (UID: \"5d494afe-0126-4460-9f0a-614f6eb4f0f1\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"
Oct 30 03:45:23.828266 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.828173    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gj46\" (UniqueName: \"kubernetes.io/projected/5d494afe-0126-4460-9f0a-614f6eb4f0f1-kube-api-access-8gj46\") pod \"korrel8r-5d45699dc4-4pv6g\" (UID: \"5d494afe-0126-4460-9f0a-614f6eb4f0f1\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"
Oct 30 03:45:23.828266 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.828221    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5d494afe-0126-4460-9f0a-614f6eb4f0f1-config\") pod \"korrel8r-5d45699dc4-4pv6g\" (UID: \"5d494afe-0126-4460-9f0a-614f6eb4f0f1\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"
Oct 30 03:45:23.828851 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.828835    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5d494afe-0126-4460-9f0a-614f6eb4f0f1-config\") pod \"korrel8r-5d45699dc4-4pv6g\" (UID: \"5d494afe-0126-4460-9f0a-614f6eb4f0f1\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"
Oct 30 03:45:23.830593 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.830572    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5d494afe-0126-4460-9f0a-614f6eb4f0f1-tls\") pod \"korrel8r-5d45699dc4-4pv6g\" (UID: \"5d494afe-0126-4460-9f0a-614f6eb4f0f1\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"
Oct 30 03:45:23.849273 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.849253    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gj46\" (UniqueName: \"kubernetes.io/projected/5d494afe-0126-4460-9f0a-614f6eb4f0f1-kube-api-access-8gj46\") pod \"korrel8r-5d45699dc4-4pv6g\" (UID: \"5d494afe-0126-4460-9f0a-614f6eb4f0f1\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"
Oct 30 03:45:23.859220 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.859199    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:23.926862 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:23.926843    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"
Oct 30 03:45:23.955128 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:45:23.955092    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:45:24.225396 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.225370    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"]
Oct 30 03:45:24.227753 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:45:24.227713    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d494afe_0126_4460_9f0a_614f6eb4f0f1.slice/crio-ae410185b971afd9eac4d203d4b96eddcc0ecf78f278a83e8dac143f6acd9a53 WatchSource:0}: Error finding container ae410185b971afd9eac4d203d4b96eddcc0ecf78f278a83e8dac143f6acd9a53: Status 404 returned error can't find the container with id ae410185b971afd9eac4d203d4b96eddcc0ecf78f278a83e8dac143f6acd9a53
Oct 30 03:45:24.288013 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.287987    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:24.291722 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.291704    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:24.311683 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.311659    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:24.334506 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.334483    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:24.360546 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.360522    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:45:24.360706 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.360607    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:45:24.360706 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.360696    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-r7h6s\""
Oct 30 03:45:24.381508 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.381490    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\""
Oct 30 03:45:24.433759 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.433738    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad640c1f-3918-4d3a-ad75-00bbea5aa002-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:24.433873 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.433765    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8nfs\" (UniqueName: \"kubernetes.io/projected/ad640c1f-3918-4d3a-ad75-00bbea5aa002-kube-api-access-x8nfs\") pod \"data-science-collector-collector-1\" (UID: \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:24.433873 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.433782    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad640c1f-3918-4d3a-ad75-00bbea5aa002-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:24.433873 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.433811    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad640c1f-3918-4d3a-ad75-00bbea5aa002-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:24.456858 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.456824    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:24.473400 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.473373    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g" event={"ID":"5d494afe-0126-4460-9f0a-614f6eb4f0f1","Type":"ContainerStarted","Data":"9ae16178987a127988b4dd055b8b2c712e08b7c5c03379b9f682f6a30e3f3cbf"}
Oct 30 03:45:24.473522 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.473433    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g" event={"ID":"5d494afe-0126-4460-9f0a-614f6eb4f0f1","Type":"ContainerStarted","Data":"ae410185b971afd9eac4d203d4b96eddcc0ecf78f278a83e8dac143f6acd9a53"}
Oct 30 03:45:24.473522 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.473467    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"
Oct 30 03:45:24.517273 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.517192    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g" podStartSLOduration=1.5171657349999998 podStartE2EDuration="1.517165735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:24.51615565 +0000 UTC m=+3805.165204071" watchObservedRunningTime="2026-10-30 03:45:24.517165735 +0000 UTC m=+3805.166214153"
Oct 30 03:45:24.534260 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.534232    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad640c1f-3918-4d3a-ad75-00bbea5aa002-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:24.534377 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.534262    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8nfs\" (UniqueName: \"kubernetes.io/projected/ad640c1f-3918-4d3a-ad75-00bbea5aa002-kube-api-access-x8nfs\") pod \"data-science-collector-collector-1\" (UID: \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:24.534377 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.534282    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad640c1f-3918-4d3a-ad75-00bbea5aa002-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:24.534494 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.534410    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad640c1f-3918-4d3a-ad75-00bbea5aa002-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:24.535042 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.535020    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad640c1f-3918-4d3a-ad75-00bbea5aa002-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:24.536981 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.536964    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad640c1f-3918-4d3a-ad75-00bbea5aa002-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:24.536981 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.536973    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad640c1f-3918-4d3a-ad75-00bbea5aa002-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:24.557768 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.557750    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8nfs\" (UniqueName: \"kubernetes.io/projected/ad640c1f-3918-4d3a-ad75-00bbea5aa002-kube-api-access-x8nfs\") pod \"data-science-collector-collector-1\" (UID: \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:24.606641 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.606621    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:24.620286 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.620261    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:24.910311 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.910284    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:24.911309 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:45:24.911283    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad640c1f_3918_4d3a_ad75_00bbea5aa002.slice/crio-6dac24a3d65e986b181a168394fd129c8f01873fbb05fe18a7a2553e1280795f WatchSource:0}: Error finding container 6dac24a3d65e986b181a168394fd129c8f01873fbb05fe18a7a2553e1280795f: Status 404 returned error can't find the container with id 6dac24a3d65e986b181a168394fd129c8f01873fbb05fe18a7a2553e1280795f
Oct 30 03:45:24.913761 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.913724    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:45:24.913867 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:24.913799    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:45:25.481832 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:25.481799    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ad640c1f-3918-4d3a-ad75-00bbea5aa002","Type":"ContainerStarted","Data":"79d343887ad449d2fb44170b2598a4af920cae568dfca8cba774043d507b8e18"}
Oct 30 03:45:25.482010 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:25.481860    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ad640c1f-3918-4d3a-ad75-00bbea5aa002","Type":"ContainerStarted","Data":"6dac24a3d65e986b181a168394fd129c8f01873fbb05fe18a7a2553e1280795f"}
Oct 30 03:45:25.524423 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:25.524382    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.524367448 podStartE2EDuration="1.524367448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:25.524140985 +0000 UTC m=+3806.173189408" watchObservedRunningTime="2026-10-30 03:45:25.524367448 +0000 UTC m=+3806.173415869"
Oct 30 03:45:26.896529 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:26.896491    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:27.029537 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.029500    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:27.229917 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.229847    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:27.353232 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.353194    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:27.467687 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.467653    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:27.513844 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.513771    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"]
Oct 30 03:45:27.517438 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.517414    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.533979 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.533935    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\""
Oct 30 03:45:27.534098 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.534060    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\""
Oct 30 03:45:27.534098 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.534076    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\""
Oct 30 03:45:27.547158 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.547124    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\""
Oct 30 03:45:27.547308 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.547268    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-pxnpq\""
Oct 30 03:45:27.547375 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.547332    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\""
Oct 30 03:45:27.547427 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.547368    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\""
Oct 30 03:45:27.548807 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.548790    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"]
Oct 30 03:45:27.663321 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.663298    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.663492 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.663328    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-serving-certs\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.663492 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.663351    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.663492 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.663424    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgx46\" (UniqueName: \"kubernetes.io/projected/1a91e694-f2f9-4965-9db1-c1327c3c252d-kube-api-access-hgx46\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.663492 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.663460    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-cabundle\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.663492 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.663480    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-tenant\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.663657 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.663500    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-rbac\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.764123 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.764065    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.764123 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.764093    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-serving-certs\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.764123 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.764114    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.764331 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.764134    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgx46\" (UniqueName: \"kubernetes.io/projected/1a91e694-f2f9-4965-9db1-c1327c3c252d-kube-api-access-hgx46\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.764331 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.764150    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-cabundle\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.764331 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.764166    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-tenant\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.764331 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.764202    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-rbac\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.765059 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.765035    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-cabundle\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.765144 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.765054    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-rbac\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.765144 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.765127    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.766901 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.766879    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.766984 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.766928    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-serving-certs\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.767025 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.766998    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-tenant\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.787103 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.787081    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgx46\" (UniqueName: \"kubernetes.io/projected/1a91e694-f2f9-4965-9db1-c1327c3c252d-kube-api-access-hgx46\") pod \"tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.794336 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.794316    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"]
Oct 30 03:45:27.799505 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.799488    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:45:27.816360 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.816341    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-distributor-mtls\""
Oct 30 03:45:27.816477 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.816423    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-dockercfg-b5l6v\""
Oct 30 03:45:27.816603 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.816587    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack\""
Oct 30 03:45:27.817791 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.817768    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"]
Oct 30 03:45:27.832797 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.832775    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:27.865543 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.865520    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brzk4\" (UniqueName: \"kubernetes.io/projected/aa0b5af3-99b2-43d4-b875-d392051f7da8-kube-api-access-brzk4\") pod \"tempo-data-science-tempostack-distributor-6cd5559486-gtqmc\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:45:27.865650 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.865556    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-6cd5559486-gtqmc\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:45:27.865650 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.865587    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-6cd5559486-gtqmc\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:45:27.865744 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.865679    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-6cd5559486-gtqmc\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:45:27.865803 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.865783    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-6cd5559486-gtqmc\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:45:27.878545 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.878513    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"]
Oct 30 03:45:27.887861 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.886838    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:27.909097 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.904165    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"]
Oct 30 03:45:27.917216 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.911582    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"]
Oct 30 03:45:27.917216 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.911699    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:27.917216 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.915980    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"gcs-secret\""
Oct 30 03:45:27.917216 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.916367    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-query-frontend-mtls\""
Oct 30 03:45:27.931252 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.931048    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"]
Oct 30 03:45:27.933040 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.931887    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"]
Oct 30 03:45:27.935623 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.935238    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:27.954804 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.954779    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"]
Oct 30 03:45:27.966831 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.966795    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-6cd5559486-gtqmc\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:45:27.966951 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.966846    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:27.966951 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.966872    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:27.966951 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.966909    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-6cd5559486-gtqmc\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:45:27.967123 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.967099    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s8nm\" (UniqueName: \"kubernetes.io/projected/d53f2afb-6b3d-4290-86f3-c663ef9734be-kube-api-access-5s8nm\") pod \"tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:27.967213 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.967136    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brzk4\" (UniqueName: \"kubernetes.io/projected/aa0b5af3-99b2-43d4-b875-d392051f7da8-kube-api-access-brzk4\") pod \"tempo-data-science-tempostack-distributor-6cd5559486-gtqmc\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:45:27.967213 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.967169    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:27.967334 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.967216    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:27.967334 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.967265    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-6cd5559486-gtqmc\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:45:27.967447 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.967338    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d53f2afb-6b3d-4290-86f3-c663ef9734be-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:27.967447 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.967373    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-6cd5559486-gtqmc\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:45:27.967743 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.967725    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-6cd5559486-gtqmc\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:45:27.967844 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.967828    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-6cd5559486-gtqmc\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:45:27.968243 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.968172    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-6cd5559486-gtqmc\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:45:27.969676 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.969653    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-6cd5559486-gtqmc\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:45:27.988088 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:27.988068    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brzk4\" (UniqueName: \"kubernetes.io/projected/aa0b5af3-99b2-43d4-b875-d392051f7da8-kube-api-access-brzk4\") pod \"tempo-data-science-tempostack-distributor-6cd5559486-gtqmc\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:45:28.016427 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.016374    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-querier-mtls\""
Oct 30 03:45:28.016427 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.016378    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-compactor-mtls\""
Oct 30 03:45:28.068102 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.068075    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/61bc4b06-9920-4266-a83e-59233486fc7b-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:28.068286 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.068120    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:28.068286 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.068238    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:28.068408 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.068286    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-84f565bdd4-c7ksn\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:28.068408 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.068362    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2glp\" (UniqueName: \"kubernetes.io/projected/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-kube-api-access-p2glp\") pod \"tempo-data-science-tempostack-querier-84f565bdd4-c7ksn\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:28.068502 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.068442    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:28.068502 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.068460    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:28.068502 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.068486    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-84f565bdd4-c7ksn\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:28.068648 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.068525    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s8nm\" (UniqueName: \"kubernetes.io/projected/d53f2afb-6b3d-4290-86f3-c663ef9734be-kube-api-access-5s8nm\") pod \"tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:28.068648 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.068558    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:28.068648 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.068602    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:28.068811 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.068725    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:28.068811 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.068751    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-84f565bdd4-c7ksn\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:28.068811 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.068771    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-84f565bdd4-c7ksn\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:28.068811 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.068796    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l68z7\" (UniqueName: \"kubernetes.io/projected/61bc4b06-9920-4266-a83e-59233486fc7b-kube-api-access-l68z7\") pod \"tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:28.069020 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.068906    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d53f2afb-6b3d-4290-86f3-c663ef9734be-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:28.069020 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.068931    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:28.069020 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.068943    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:28.069020 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.068980    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-conf\") pod \"tempo-data-science-tempostack-querier-84f565bdd4-c7ksn\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:28.069443 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.069420    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:28.069515 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.069455    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:28.071203 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.071164    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:28.071421 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.071400    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d53f2afb-6b3d-4290-86f3-c663ef9734be-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:28.095474 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.095453    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s8nm\" (UniqueName: \"kubernetes.io/projected/d53f2afb-6b3d-4290-86f3-c663ef9734be-kube-api-access-5s8nm\") pod \"tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:28.112403 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.112385    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:45:28.170736 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.170647    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:28.170736 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.170707    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-84f565bdd4-c7ksn\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:28.170932 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.170739    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2glp\" (UniqueName: \"kubernetes.io/projected/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-kube-api-access-p2glp\") pod \"tempo-data-science-tempostack-querier-84f565bdd4-c7ksn\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:28.170932 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.170794    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-84f565bdd4-c7ksn\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:28.170932 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.170850    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:28.170932 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.170867    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-84f565bdd4-c7ksn\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:28.170932 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.170884    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-84f565bdd4-c7ksn\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:28.170932 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.170898    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l68z7\" (UniqueName: \"kubernetes.io/projected/61bc4b06-9920-4266-a83e-59233486fc7b-kube-api-access-l68z7\") pod \"tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:28.170932 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.170928    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:28.171337 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.170957    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-conf\") pod \"tempo-data-science-tempostack-querier-84f565bdd4-c7ksn\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:28.171337 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.170981    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/61bc4b06-9920-4266-a83e-59233486fc7b-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:28.171337 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.171001    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:28.171766 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.171739    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:28.172064 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.172039    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:28.172726 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.172702    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-84f565bdd4-c7ksn\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:28.174683 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.173531    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-84f565bdd4-c7ksn\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:28.174683 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.174035    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:28.174957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.174917    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-conf\") pod \"tempo-data-science-tempostack-querier-84f565bdd4-c7ksn\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:28.175960 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.175934    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-84f565bdd4-c7ksn\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:28.176414 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.176393    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:28.176832 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.176808    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/61bc4b06-9920-4266-a83e-59233486fc7b-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:28.177550 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.177529    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-84f565bdd4-c7ksn\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:28.187705 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.187681    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"]
Oct 30 03:45:28.188850 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:45:28.188828    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a91e694_f2f9_4965_9db1_c1327c3c252d.slice/crio-74acca4579229eeae2a0a4ec8f656798dd6330e4b7c069dfa5b00962386401d8 WatchSource:0}: Error finding container 74acca4579229eeae2a0a4ec8f656798dd6330e4b7c069dfa5b00962386401d8: Status 404 returned error can't find the container with id 74acca4579229eeae2a0a4ec8f656798dd6330e4b7c069dfa5b00962386401d8
Oct 30 03:45:28.204435 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.204410    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2glp\" (UniqueName: \"kubernetes.io/projected/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-kube-api-access-p2glp\") pod \"tempo-data-science-tempostack-querier-84f565bdd4-c7ksn\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:28.205499 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.205478    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l68z7\" (UniqueName: \"kubernetes.io/projected/61bc4b06-9920-4266-a83e-59233486fc7b-kube-api-access-l68z7\") pod \"tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:28.213422 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.213401    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:28.230017 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.229998    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:28.254687 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.254640    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:28.406690 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.406643    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"]
Oct 30 03:45:28.409051 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:45:28.409024    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0b5af3_99b2_43d4_b875_d392051f7da8.slice/crio-1fd70eedc1b4b4df7ac50342fbabfd3fdab2eb349b7e892d620072961983f46e WatchSource:0}: Error finding container 1fd70eedc1b4b4df7ac50342fbabfd3fdab2eb349b7e892d620072961983f46e: Status 404 returned error can't find the container with id 1fd70eedc1b4b4df7ac50342fbabfd3fdab2eb349b7e892d620072961983f46e
Oct 30 03:45:28.504167 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.504141    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr" event={"ID":"1a91e694-f2f9-4965-9db1-c1327c3c252d","Type":"ContainerStarted","Data":"ee942bf8a488985529bd17d9d4657501bb7acf225023a99f04175cd46d6cda3b"}
Oct 30 03:45:28.504310 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.504210    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr" event={"ID":"1a91e694-f2f9-4965-9db1-c1327c3c252d","Type":"ContainerStarted","Data":"74acca4579229eeae2a0a4ec8f656798dd6330e4b7c069dfa5b00962386401d8"}
Oct 30 03:45:28.505289 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.505268    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc" event={"ID":"aa0b5af3-99b2-43d4-b875-d392051f7da8","Type":"ContainerStarted","Data":"1fd70eedc1b4b4df7ac50342fbabfd3fdab2eb349b7e892d620072961983f46e"}
Oct 30 03:45:28.510807 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.510782    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"]
Oct 30 03:45:28.511961 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:45:28.511937    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd53f2afb_6b3d_4290_86f3_c663ef9734be.slice/crio-d85292650f5d63caee3c26a7f3679901e181e10478dfb4aeb589f6c77a995dc6 WatchSource:0}: Error finding container d85292650f5d63caee3c26a7f3679901e181e10478dfb4aeb589f6c77a995dc6: Status 404 returned error can't find the container with id d85292650f5d63caee3c26a7f3679901e181e10478dfb4aeb589f6c77a995dc6
Oct 30 03:45:28.535400 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.535340    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"]
Oct 30 03:45:28.536533 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:45:28.536507    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1524f4c0_dd9a_4c2b_811e_4e10af5a62fd.slice/crio-5c6a20d8b9875c741f14a4afcc2d98eb0990bf94c6caf41598e6fd9ad5bd1003 WatchSource:0}: Error finding container 5c6a20d8b9875c741f14a4afcc2d98eb0990bf94c6caf41598e6fd9ad5bd1003: Status 404 returned error can't find the container with id 5c6a20d8b9875c741f14a4afcc2d98eb0990bf94c6caf41598e6fd9ad5bd1003
Oct 30 03:45:28.573249 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:28.573227    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"]
Oct 30 03:45:28.574229 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:45:28.574208    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61bc4b06_9920_4266_a83e_59233486fc7b.slice/crio-12b37036922c5a9776ee87b3a109539a0df5bb9e145d59cad695a92ab24ac4a5 WatchSource:0}: Error finding container 12b37036922c5a9776ee87b3a109539a0df5bb9e145d59cad695a92ab24ac4a5: Status 404 returned error can't find the container with id 12b37036922c5a9776ee87b3a109539a0df5bb9e145d59cad695a92ab24ac4a5
Oct 30 03:45:29.518124 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:29.518084    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl" event={"ID":"61bc4b06-9920-4266-a83e-59233486fc7b","Type":"ContainerStarted","Data":"12b37036922c5a9776ee87b3a109539a0df5bb9e145d59cad695a92ab24ac4a5"}
Oct 30 03:45:29.519952 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:29.519923    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf" event={"ID":"d53f2afb-6b3d-4290-86f3-c663ef9734be","Type":"ContainerStarted","Data":"d85292650f5d63caee3c26a7f3679901e181e10478dfb4aeb589f6c77a995dc6"}
Oct 30 03:45:29.522898 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:29.522870    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr" event={"ID":"1a91e694-f2f9-4965-9db1-c1327c3c252d","Type":"ContainerStarted","Data":"274fd27571da949c91c47b5d977d7f82bf1fff8db5257877465207a04dec9748"}
Oct 30 03:45:29.523993 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:29.523972    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:29.524087 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:29.524027    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:29.525834 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:29.525630    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr" podUID="1a91e694-f2f9-4965-9db1-c1327c3c252d" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.74:8083/ready\": dial tcp 10.132.0.74:8083: connect: connection refused"
Oct 30 03:45:29.525834 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:29.525638    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr" podUID="1a91e694-f2f9-4965-9db1-c1327c3c252d" containerName="tempo-gateway" probeResult="failure" output="Get \"https://10.132.0.74:8081/ready\": dial tcp 10.132.0.74:8081: connect: connection refused"
Oct 30 03:45:29.528654 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:29.527893    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn" event={"ID":"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd","Type":"ContainerStarted","Data":"5c6a20d8b9875c741f14a4afcc2d98eb0990bf94c6caf41598e6fd9ad5bd1003"}
Oct 30 03:45:30.696369 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:30.695623    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:30.769685 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:30.769611    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr" podStartSLOduration=3.769591318 podStartE2EDuration="3.769591318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:29.598135126 +0000 UTC m=+3810.247183548" watchObservedRunningTime="2026-10-30 03:45:30.769591318 +0000 UTC m=+3811.418639742"
Oct 30 03:45:30.895593 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:30.895564    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:31.174973 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:31.174941    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"]
Oct 30 03:45:32.548670 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:32.548630    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf" event={"ID":"d53f2afb-6b3d-4290-86f3-c663ef9734be","Type":"ContainerStarted","Data":"3ceff0c4c9ac1a5141dffe6bf02eb1aab2cf22301c98e1668193c4b4eb50fda4"}
Oct 30 03:45:32.549126 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:32.548795    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:32.550194 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:32.550161    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn" event={"ID":"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd","Type":"ContainerStarted","Data":"2248957b195cc1db3244ded85e8f25bbb750fbc8357d0874e7e35f2102db1173"}
Oct 30 03:45:32.550300 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:32.550224    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:32.551612 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:32.551591    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc" event={"ID":"aa0b5af3-99b2-43d4-b875-d392051f7da8","Type":"ContainerStarted","Data":"2edf6f154fe41047872a96fa4e505ef8f70f60495bf22485183d7c95e0954275"}
Oct 30 03:45:32.551696 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:32.551638    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:45:32.553159 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:32.553106    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl" event={"ID":"61bc4b06-9920-4266-a83e-59233486fc7b","Type":"ContainerStarted","Data":"cff831dcf72d5e524d80ad1fd83c333a394d96be4c529047237f18f58fe0d670"}
Oct 30 03:45:32.553279 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:32.553216    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:45:32.553279 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:32.553228    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr" podUID="1a91e694-f2f9-4965-9db1-c1327c3c252d" containerName="tempo-gateway" containerID="cri-o://ee942bf8a488985529bd17d9d4657501bb7acf225023a99f04175cd46d6cda3b" gracePeriod=30
Oct 30 03:45:32.553279 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:32.553254    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr" podUID="1a91e694-f2f9-4965-9db1-c1327c3c252d" containerName="tempo-gateway-opa" containerID="cri-o://274fd27571da949c91c47b5d977d7f82bf1fff8db5257877465207a04dec9748" gracePeriod=30
Oct 30 03:45:32.591315 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:32.591268    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf" podStartSLOduration=2.163135338 podStartE2EDuration="5.591255452s" totalImagesPullingTime="3.428120114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:32.58959058 +0000 UTC m=+3813.238639001" watchObservedRunningTime="2026-10-30 03:45:32.591255452 +0000 UTC m=+3813.240303872"
Oct 30 03:45:32.675081 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:32.675036    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn" podStartSLOduration=2.286705188 podStartE2EDuration="5.675023724s" totalImagesPullingTime="3.388318536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:32.673065052 +0000 UTC m=+3813.322113474" watchObservedRunningTime="2026-10-30 03:45:32.675023724 +0000 UTC m=+3813.324072146"
Oct 30 03:45:32.718658 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:32.718612    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl" podStartSLOduration=2.368463856 podStartE2EDuration="5.718595498s" totalImagesPullingTime="3.350131642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:32.71835646 +0000 UTC m=+3813.367404893" watchObservedRunningTime="2026-10-30 03:45:32.718595498 +0000 UTC m=+3813.367643920"
Oct 30 03:45:32.760727 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:32.760673    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc" podStartSLOduration=2.246690114 podStartE2EDuration="5.760659103s" totalImagesPullingTime="3.513968989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:32.760618808 +0000 UTC m=+3813.409667231" watchObservedRunningTime="2026-10-30 03:45:32.760659103 +0000 UTC m=+3813.409707524"
Oct 30 03:45:33.559756 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:33.559724    4960 generic.go:356] "Generic (PLEG): container finished" podID="1a91e694-f2f9-4965-9db1-c1327c3c252d" containerID="274fd27571da949c91c47b5d977d7f82bf1fff8db5257877465207a04dec9748" exitCode=0
Oct 30 03:45:33.560106 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:33.559803    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr" event={"ID":"1a91e694-f2f9-4965-9db1-c1327c3c252d","Type":"ContainerDied","Data":"274fd27571da949c91c47b5d977d7f82bf1fff8db5257877465207a04dec9748"}
Oct 30 03:45:33.810923 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:33.810868    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:33.924930 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:33.924902    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-tenant\" (UniqueName: \"kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-tenant\") pod \"1a91e694-f2f9-4965-9db1-c1327c3c252d\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") "
Oct 30 03:45:33.925102 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:33.924985    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-tempo-data-science-tempostack-gateway-mtls\") pod \"1a91e694-f2f9-4965-9db1-c1327c3c252d\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") "
Oct 30 03:45:33.925102 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:33.925020    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a91e694-f2f9-4965-9db1-c1327c3c252d-kube-api-access-hgx46\" (UniqueName: \"kubernetes.io/projected/1a91e694-f2f9-4965-9db1-c1327c3c252d-kube-api-access-hgx46\") pod \"1a91e694-f2f9-4965-9db1-c1327c3c252d\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") "
Oct 30 03:45:33.925102 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:33.925071    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-serving-certs\" (UniqueName: \"kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-serving-certs\") pod \"1a91e694-f2f9-4965-9db1-c1327c3c252d\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") "
Oct 30 03:45:33.925298 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:33.925101    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-tempo-data-science-tempostack-ca-bundle\") pod \"1a91e694-f2f9-4965-9db1-c1327c3c252d\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") "
Oct 30 03:45:33.925298 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:33.925142    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-rbac\" (UniqueName: \"kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-rbac\") pod \"1a91e694-f2f9-4965-9db1-c1327c3c252d\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") "
Oct 30 03:45:33.925298 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:33.925203    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-cabundle\" (UniqueName: \"kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-cabundle\") pod \"1a91e694-f2f9-4965-9db1-c1327c3c252d\" (UID: \"1a91e694-f2f9-4965-9db1-c1327c3c252d\") "
Oct 30 03:45:33.926006 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:33.925866    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-cabundle" pod "1a91e694-f2f9-4965-9db1-c1327c3c252d" (UID: "1a91e694-f2f9-4965-9db1-c1327c3c252d"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:33.926006 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:33.925871    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-rbac" pod "1a91e694-f2f9-4965-9db1-c1327c3c252d" (UID: "1a91e694-f2f9-4965-9db1-c1327c3c252d"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:33.926006 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:33.925986    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-tempo-data-science-tempostack-ca-bundle" pod "1a91e694-f2f9-4965-9db1-c1327c3c252d" (UID: "1a91e694-f2f9-4965-9db1-c1327c3c252d"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:33.927659 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:33.927633    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a91e694-f2f9-4965-9db1-c1327c3c252d-kube-api-access-hgx46" pod "1a91e694-f2f9-4965-9db1-c1327c3c252d" (UID: "1a91e694-f2f9-4965-9db1-c1327c3c252d"). InnerVolumeSpecName "kube-api-access-hgx46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:33.927916 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:33.927902    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-tempo-data-science-tempostack-gateway-mtls" pod "1a91e694-f2f9-4965-9db1-c1327c3c252d" (UID: "1a91e694-f2f9-4965-9db1-c1327c3c252d"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:33.927984 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:33.927968    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-serving-certs" pod "1a91e694-f2f9-4965-9db1-c1327c3c252d" (UID: "1a91e694-f2f9-4965-9db1-c1327c3c252d"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:33.932864 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:33.932842    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-tenant" pod "1a91e694-f2f9-4965-9db1-c1327c3c252d" (UID: "1a91e694-f2f9-4965-9db1-c1327c3c252d"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:34.026308 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:34.026285    4960 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-tenant\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:45:34.026308 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:34.026308    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:45:34.026426 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:34.026318    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgx46\" (UniqueName: \"kubernetes.io/projected/1a91e694-f2f9-4965-9db1-c1327c3c252d-kube-api-access-hgx46\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:45:34.026426 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:34.026328    4960 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/1a91e694-f2f9-4965-9db1-c1327c3c252d-serving-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:45:34.026426 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:34.026339    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:45:34.026426 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:34.026353    4960 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-rbac\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:45:34.026426 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:34.026361    4960 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/1a91e694-f2f9-4965-9db1-c1327c3c252d-cabundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:45:34.571911 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:34.571880    4960 generic.go:356] "Generic (PLEG): container finished" podID="1a91e694-f2f9-4965-9db1-c1327c3c252d" containerID="ee942bf8a488985529bd17d9d4657501bb7acf225023a99f04175cd46d6cda3b" exitCode=0
Oct 30 03:45:34.572321 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:34.571921    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr" event={"ID":"1a91e694-f2f9-4965-9db1-c1327c3c252d","Type":"ContainerDied","Data":"ee942bf8a488985529bd17d9d4657501bb7acf225023a99f04175cd46d6cda3b"}
Oct 30 03:45:34.572321 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:34.571947    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"
Oct 30 03:45:34.572321 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:34.571988    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr" event={"ID":"1a91e694-f2f9-4965-9db1-c1327c3c252d","Type":"ContainerDied","Data":"74acca4579229eeae2a0a4ec8f656798dd6330e4b7c069dfa5b00962386401d8"}
Oct 30 03:45:34.572321 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:34.572008    4960 scope.go:122] "RemoveContainer" containerID="274fd27571da949c91c47b5d977d7f82bf1fff8db5257877465207a04dec9748"
Oct 30 03:45:34.581338 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:34.581317    4960 scope.go:122] "RemoveContainer" containerID="ee942bf8a488985529bd17d9d4657501bb7acf225023a99f04175cd46d6cda3b"
Oct 30 03:45:34.589085 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:34.589070    4960 scope.go:122] "RemoveContainer" containerID="274fd27571da949c91c47b5d977d7f82bf1fff8db5257877465207a04dec9748"
Oct 30 03:45:34.589394 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:45:34.589375    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"274fd27571da949c91c47b5d977d7f82bf1fff8db5257877465207a04dec9748\": container with ID starting with 274fd27571da949c91c47b5d977d7f82bf1fff8db5257877465207a04dec9748 not found: ID does not exist" containerID="274fd27571da949c91c47b5d977d7f82bf1fff8db5257877465207a04dec9748"
Oct 30 03:45:34.589459 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:34.589402    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"274fd27571da949c91c47b5d977d7f82bf1fff8db5257877465207a04dec9748"} err="failed to get container status \"274fd27571da949c91c47b5d977d7f82bf1fff8db5257877465207a04dec9748\": rpc error: code = NotFound desc = could not find container \"274fd27571da949c91c47b5d977d7f82bf1fff8db5257877465207a04dec9748\": container with ID starting with 274fd27571da949c91c47b5d977d7f82bf1fff8db5257877465207a04dec9748 not found: ID does not exist"
Oct 30 03:45:34.589459 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:34.589420    4960 scope.go:122] "RemoveContainer" containerID="ee942bf8a488985529bd17d9d4657501bb7acf225023a99f04175cd46d6cda3b"
Oct 30 03:45:34.589660 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:45:34.589641    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee942bf8a488985529bd17d9d4657501bb7acf225023a99f04175cd46d6cda3b\": container with ID starting with ee942bf8a488985529bd17d9d4657501bb7acf225023a99f04175cd46d6cda3b not found: ID does not exist" containerID="ee942bf8a488985529bd17d9d4657501bb7acf225023a99f04175cd46d6cda3b"
Oct 30 03:45:34.589712 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:34.589669    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee942bf8a488985529bd17d9d4657501bb7acf225023a99f04175cd46d6cda3b"} err="failed to get container status \"ee942bf8a488985529bd17d9d4657501bb7acf225023a99f04175cd46d6cda3b\": rpc error: code = NotFound desc = could not find container \"ee942bf8a488985529bd17d9d4657501bb7acf225023a99f04175cd46d6cda3b\": container with ID starting with ee942bf8a488985529bd17d9d4657501bb7acf225023a99f04175cd46d6cda3b not found: ID does not exist"
Oct 30 03:45:34.642636 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:34.642610    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"]
Oct 30 03:45:34.656530 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:34.656502    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54bc7948dc-nzvsr"]
Oct 30 03:45:35.483948 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:35.483921    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"
Oct 30 03:45:35.502555 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:35.502528    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a91e694-f2f9-4965-9db1-c1327c3c252d" path="/var/lib/kubelet/pods/1a91e694-f2f9-4965-9db1-c1327c3c252d/volumes"
Oct 30 03:45:53.567153 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:53.567121    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:45:53.567585 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:53.567306    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:45:53.567585 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:53.567451    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl" podUID="61bc4b06-9920-4266-a83e-59233486fc7b" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:45:53.697039 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:53.697008    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:45:53.953958 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:45:53.953882    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:45:56.834997 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:45:56.834961    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:03.566219 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:03.566156    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl" podUID="61bc4b06-9920-4266-a83e-59233486fc7b" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:46:13.564740 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:13.564702    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl" podUID="61bc4b06-9920-4266-a83e-59233486fc7b" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:46:23.565733 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:23.565678    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl" podUID="61bc4b06-9920-4266-a83e-59233486fc7b" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:46:23.953047 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:46:23.952952    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:46:33.566029 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:33.566002    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:46:36.815218 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:36.815166    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:36.923328 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:36.923291    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:37.310302 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.310224    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:46:37.310460 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.310410    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="ad640c1f-3918-4d3a-ad75-00bbea5aa002" containerName="otc-container" containerID="cri-o://79d343887ad449d2fb44170b2598a4af920cae568dfca8cba774043d507b8e18" gracePeriod=30
Oct 30 03:46:37.454794 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.454765    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:37.566692 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.566672    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:46:37.578006 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.577978    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:37.705685 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.705652    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad640c1f-3918-4d3a-ad75-00bbea5aa002-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad640c1f-3918-4d3a-ad75-00bbea5aa002-otlp-tls-certs\") pod \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\" (UID: \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\") "
Oct 30 03:46:37.705685 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.705688    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad640c1f-3918-4d3a-ad75-00bbea5aa002-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad640c1f-3918-4d3a-ad75-00bbea5aa002-monitoring-tls-certs\") pod \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\" (UID: \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\") "
Oct 30 03:46:37.705926 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.705714    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad640c1f-3918-4d3a-ad75-00bbea5aa002-kube-api-access-x8nfs\" (UniqueName: \"kubernetes.io/projected/ad640c1f-3918-4d3a-ad75-00bbea5aa002-kube-api-access-x8nfs\") pod \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\" (UID: \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\") "
Oct 30 03:46:37.705926 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.705796    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad640c1f-3918-4d3a-ad75-00bbea5aa002-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad640c1f-3918-4d3a-ad75-00bbea5aa002-otc-internal\") pod \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\" (UID: \"ad640c1f-3918-4d3a-ad75-00bbea5aa002\") "
Oct 30 03:46:37.706447 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.706367    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad640c1f-3918-4d3a-ad75-00bbea5aa002-otc-internal" pod "ad640c1f-3918-4d3a-ad75-00bbea5aa002" (UID: "ad640c1f-3918-4d3a-ad75-00bbea5aa002"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:37.707944 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.707921    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad640c1f-3918-4d3a-ad75-00bbea5aa002-monitoring-tls-certs" pod "ad640c1f-3918-4d3a-ad75-00bbea5aa002" (UID: "ad640c1f-3918-4d3a-ad75-00bbea5aa002"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:46:37.708030 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.707987    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad640c1f-3918-4d3a-ad75-00bbea5aa002-kube-api-access-x8nfs" pod "ad640c1f-3918-4d3a-ad75-00bbea5aa002" (UID: "ad640c1f-3918-4d3a-ad75-00bbea5aa002"). InnerVolumeSpecName "kube-api-access-x8nfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:46:37.708030 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.707991    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad640c1f-3918-4d3a-ad75-00bbea5aa002-otlp-tls-certs" pod "ad640c1f-3918-4d3a-ad75-00bbea5aa002" (UID: "ad640c1f-3918-4d3a-ad75-00bbea5aa002"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:46:37.807397 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.807375    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad640c1f-3918-4d3a-ad75-00bbea5aa002-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:46:37.807397 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.807394    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad640c1f-3918-4d3a-ad75-00bbea5aa002-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:46:37.807525 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.807404    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8nfs\" (UniqueName: \"kubernetes.io/projected/ad640c1f-3918-4d3a-ad75-00bbea5aa002-kube-api-access-x8nfs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:46:37.807525 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.807413    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad640c1f-3918-4d3a-ad75-00bbea5aa002-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:46:37.877541 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.877471    4960 generic.go:356] "Generic (PLEG): container finished" podID="ad640c1f-3918-4d3a-ad75-00bbea5aa002" containerID="79d343887ad449d2fb44170b2598a4af920cae568dfca8cba774043d507b8e18" exitCode=0
Oct 30 03:46:37.877541 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.877522    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ad640c1f-3918-4d3a-ad75-00bbea5aa002","Type":"ContainerDied","Data":"79d343887ad449d2fb44170b2598a4af920cae568dfca8cba774043d507b8e18"}
Oct 30 03:46:37.877994 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.877540    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:46:37.877994 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.877583    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ad640c1f-3918-4d3a-ad75-00bbea5aa002","Type":"ContainerDied","Data":"6dac24a3d65e986b181a168394fd129c8f01873fbb05fe18a7a2553e1280795f"}
Oct 30 03:46:37.877994 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.877603    4960 scope.go:122] "RemoveContainer" containerID="79d343887ad449d2fb44170b2598a4af920cae568dfca8cba774043d507b8e18"
Oct 30 03:46:37.886309 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.886294    4960 scope.go:122] "RemoveContainer" containerID="79d343887ad449d2fb44170b2598a4af920cae568dfca8cba774043d507b8e18"
Oct 30 03:46:37.886680 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:46:37.886662    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d343887ad449d2fb44170b2598a4af920cae568dfca8cba774043d507b8e18\": container with ID starting with 79d343887ad449d2fb44170b2598a4af920cae568dfca8cba774043d507b8e18 not found: ID does not exist" containerID="79d343887ad449d2fb44170b2598a4af920cae568dfca8cba774043d507b8e18"
Oct 30 03:46:37.886736 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.886686    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d343887ad449d2fb44170b2598a4af920cae568dfca8cba774043d507b8e18"} err="failed to get container status \"79d343887ad449d2fb44170b2598a4af920cae568dfca8cba774043d507b8e18\": rpc error: code = NotFound desc = could not find container \"79d343887ad449d2fb44170b2598a4af920cae568dfca8cba774043d507b8e18\": container with ID starting with 79d343887ad449d2fb44170b2598a4af920cae568dfca8cba774043d507b8e18 not found: ID does not exist"
Oct 30 03:46:37.931754 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.931729    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:46:37.935937 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.935910    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:46:37.974706 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.974682    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:46:37.975012 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.975001    4960 cpu_manager.go:414] "containerMap: removing container" podUID="ad640c1f-3918-4d3a-ad75-00bbea5aa002" containerName="otc-container"
Oct 30 03:46:37.975060 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.975014    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad640c1f-3918-4d3a-ad75-00bbea5aa002" containerName="otc-container"
Oct 30 03:46:37.975060 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.975034    4960 cpu_manager.go:414] "containerMap: removing container" podUID="1a91e694-f2f9-4965-9db1-c1327c3c252d" containerName="tempo-gateway"
Oct 30 03:46:37.975060 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.975039    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a91e694-f2f9-4965-9db1-c1327c3c252d" containerName="tempo-gateway"
Oct 30 03:46:37.975060 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.975058    4960 cpu_manager.go:414] "containerMap: removing container" podUID="1a91e694-f2f9-4965-9db1-c1327c3c252d" containerName="tempo-gateway-opa"
Oct 30 03:46:37.975200 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.975065    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a91e694-f2f9-4965-9db1-c1327c3c252d" containerName="tempo-gateway-opa"
Oct 30 03:46:37.975200 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.975120    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a91e694-f2f9-4965-9db1-c1327c3c252d" containerName="tempo-gateway-opa"
Oct 30 03:46:37.975200 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.975129    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad640c1f-3918-4d3a-ad75-00bbea5aa002" containerName="otc-container"
Oct 30 03:46:37.975200 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.975137    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a91e694-f2f9-4965-9db1-c1327c3c252d" containerName="tempo-gateway"
Oct 30 03:46:37.979438 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.979420    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:46:37.997299 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.997260    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:46:37.997438 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.997336    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:46:37.997438 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.997397    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-r7h6s\""
Oct 30 03:46:37.997438 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.997402    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\""
Oct 30 03:46:37.998663 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:37.998642    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:46:38.109799 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:38.109775    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4f816b-930c-4909-b51a-d563b774b4ef-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cd4f816b-930c-4909-b51a-d563b774b4ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:46:38.109929 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:38.109815    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cd4f816b-930c-4909-b51a-d563b774b4ef-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"cd4f816b-930c-4909-b51a-d563b774b4ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:46:38.109929 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:38.109919    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g6tr\" (UniqueName: \"kubernetes.io/projected/cd4f816b-930c-4909-b51a-d563b774b4ef-kube-api-access-6g6tr\") pod \"data-science-collector-collector-1\" (UID: \"cd4f816b-930c-4909-b51a-d563b774b4ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:46:38.110006 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:38.109947    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4f816b-930c-4909-b51a-d563b774b4ef-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cd4f816b-930c-4909-b51a-d563b774b4ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:46:38.211236 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:38.211165    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g6tr\" (UniqueName: \"kubernetes.io/projected/cd4f816b-930c-4909-b51a-d563b774b4ef-kube-api-access-6g6tr\") pod \"data-science-collector-collector-1\" (UID: \"cd4f816b-930c-4909-b51a-d563b774b4ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:46:38.211236 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:38.211229    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4f816b-930c-4909-b51a-d563b774b4ef-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cd4f816b-930c-4909-b51a-d563b774b4ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:46:38.211368 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:38.211262    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4f816b-930c-4909-b51a-d563b774b4ef-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cd4f816b-930c-4909-b51a-d563b774b4ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:46:38.211368 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:38.211299    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cd4f816b-930c-4909-b51a-d563b774b4ef-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"cd4f816b-930c-4909-b51a-d563b774b4ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:46:38.212022 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:38.211994    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cd4f816b-930c-4909-b51a-d563b774b4ef-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"cd4f816b-930c-4909-b51a-d563b774b4ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:46:38.213784 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:38.213760    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4f816b-930c-4909-b51a-d563b774b4ef-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cd4f816b-930c-4909-b51a-d563b774b4ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:46:38.213878 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:38.213819    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4f816b-930c-4909-b51a-d563b774b4ef-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cd4f816b-930c-4909-b51a-d563b774b4ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:46:38.232260 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:38.232238    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g6tr\" (UniqueName: \"kubernetes.io/projected/cd4f816b-930c-4909-b51a-d563b774b4ef-kube-api-access-6g6tr\") pod \"data-science-collector-collector-1\" (UID: \"cd4f816b-930c-4909-b51a-d563b774b4ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:46:38.291911 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:38.291886    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:46:38.584372 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:38.584346    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:46:38.586065 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:46:38.586035    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4f816b_930c_4909_b51a_d563b774b4ef.slice/crio-65934e00221d621e5f97b07b34d7e48bb3ad03cf76abbea8831a4d9d6700e3e5 WatchSource:0}: Error finding container 65934e00221d621e5f97b07b34d7e48bb3ad03cf76abbea8831a4d9d6700e3e5: Status 404 returned error can't find the container with id 65934e00221d621e5f97b07b34d7e48bb3ad03cf76abbea8831a4d9d6700e3e5
Oct 30 03:46:38.588416 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:38.588379    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:46:38.588519 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:38.588453    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:46:38.885131 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:38.885047    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cd4f816b-930c-4909-b51a-d563b774b4ef","Type":"ContainerStarted","Data":"15ec6bd293d086e075b28efb1653abe48548ee6fba6eab0b8b79385ddee73272"}
Oct 30 03:46:38.885131 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:38.885103    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cd4f816b-930c-4909-b51a-d563b774b4ef","Type":"ContainerStarted","Data":"65934e00221d621e5f97b07b34d7e48bb3ad03cf76abbea8831a4d9d6700e3e5"}
Oct 30 03:46:38.928641 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:38.928564    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.928552007 podStartE2EDuration="1.928552007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:38.927022174 +0000 UTC m=+3879.576070597" watchObservedRunningTime="2026-10-30 03:46:38.928552007 +0000 UTC m=+3879.577600427"
Oct 30 03:46:39.506327 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:39.506293    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad640c1f-3918-4d3a-ad75-00bbea5aa002" path="/var/lib/kubelet/pods/ad640c1f-3918-4d3a-ad75-00bbea5aa002/volumes"
Oct 30 03:46:53.954358 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:46:53.954317    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:46:56.960088 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:56.960049    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:59.883364 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:59.883325    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:46:59.887393 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:59.887369    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:46:59.889808 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:59.889773    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:46:59.893759 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:46:59.893738    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:47:10.152444 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.152410    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"]
Oct 30 03:47:10.152925 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.152597    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g" podUID="5d494afe-0126-4460-9f0a-614f6eb4f0f1" containerName="korrel8r" containerID="cri-o://9ae16178987a127988b4dd055b8b2c712e08b7c5c03379b9f682f6a30e3f3cbf" gracePeriod=30
Oct 30 03:47:10.410030 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.409977    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"
Oct 30 03:47:10.531154 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.531118    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:47:10.531713 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.531677    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="cd4f816b-930c-4909-b51a-d563b774b4ef" containerName="otc-container" containerID="cri-o://15ec6bd293d086e075b28efb1653abe48548ee6fba6eab0b8b79385ddee73272" gracePeriod=30
Oct 30 03:47:10.563644 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.563615    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d494afe-0126-4460-9f0a-614f6eb4f0f1-kube-api-access-8gj46\" (UniqueName: \"kubernetes.io/projected/5d494afe-0126-4460-9f0a-614f6eb4f0f1-kube-api-access-8gj46\") pod \"5d494afe-0126-4460-9f0a-614f6eb4f0f1\" (UID: \"5d494afe-0126-4460-9f0a-614f6eb4f0f1\") "
Oct 30 03:47:10.563816 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.563689    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d494afe-0126-4460-9f0a-614f6eb4f0f1-tls\" (UniqueName: \"kubernetes.io/secret/5d494afe-0126-4460-9f0a-614f6eb4f0f1-tls\") pod \"5d494afe-0126-4460-9f0a-614f6eb4f0f1\" (UID: \"5d494afe-0126-4460-9f0a-614f6eb4f0f1\") "
Oct 30 03:47:10.563816 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.563753    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d494afe-0126-4460-9f0a-614f6eb4f0f1-config\" (UniqueName: \"kubernetes.io/configmap/5d494afe-0126-4460-9f0a-614f6eb4f0f1-config\") pod \"5d494afe-0126-4460-9f0a-614f6eb4f0f1\" (UID: \"5d494afe-0126-4460-9f0a-614f6eb4f0f1\") "
Oct 30 03:47:10.564381 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.564349    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d494afe-0126-4460-9f0a-614f6eb4f0f1-config" pod "5d494afe-0126-4460-9f0a-614f6eb4f0f1" (UID: "5d494afe-0126-4460-9f0a-614f6eb4f0f1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:10.566089 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.566066    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d494afe-0126-4460-9f0a-614f6eb4f0f1-tls" pod "5d494afe-0126-4460-9f0a-614f6eb4f0f1" (UID: "5d494afe-0126-4460-9f0a-614f6eb4f0f1"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:10.566328 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.566308    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d494afe-0126-4460-9f0a-614f6eb4f0f1-kube-api-access-8gj46" pod "5d494afe-0126-4460-9f0a-614f6eb4f0f1" (UID: "5d494afe-0126-4460-9f0a-614f6eb4f0f1"). InnerVolumeSpecName "kube-api-access-8gj46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:47:10.595026 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.595004    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:10.664647 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.664591    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5d494afe-0126-4460-9f0a-614f6eb4f0f1-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:10.664647 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.664613    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5d494afe-0126-4460-9f0a-614f6eb4f0f1-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:10.664647 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.664623    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gj46\" (UniqueName: \"kubernetes.io/projected/5d494afe-0126-4460-9f0a-614f6eb4f0f1-kube-api-access-8gj46\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:10.785224 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.785201    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:10.966969 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.966892    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd4f816b-930c-4909-b51a-d563b774b4ef-kube-api-access-6g6tr\" (UniqueName: \"kubernetes.io/projected/cd4f816b-930c-4909-b51a-d563b774b4ef-kube-api-access-6g6tr\") pod \"cd4f816b-930c-4909-b51a-d563b774b4ef\" (UID: \"cd4f816b-930c-4909-b51a-d563b774b4ef\") "
Oct 30 03:47:10.966969 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.966949    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd4f816b-930c-4909-b51a-d563b774b4ef-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4f816b-930c-4909-b51a-d563b774b4ef-monitoring-tls-certs\") pod \"cd4f816b-930c-4909-b51a-d563b774b4ef\" (UID: \"cd4f816b-930c-4909-b51a-d563b774b4ef\") "
Oct 30 03:47:10.967166 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.967002    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd4f816b-930c-4909-b51a-d563b774b4ef-otc-internal\" (UniqueName: \"kubernetes.io/configmap/cd4f816b-930c-4909-b51a-d563b774b4ef-otc-internal\") pod \"cd4f816b-930c-4909-b51a-d563b774b4ef\" (UID: \"cd4f816b-930c-4909-b51a-d563b774b4ef\") "
Oct 30 03:47:10.967166 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.967057    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd4f816b-930c-4909-b51a-d563b774b4ef-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4f816b-930c-4909-b51a-d563b774b4ef-otlp-tls-certs\") pod \"cd4f816b-930c-4909-b51a-d563b774b4ef\" (UID: \"cd4f816b-930c-4909-b51a-d563b774b4ef\") "
Oct 30 03:47:10.967372 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.967339    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd4f816b-930c-4909-b51a-d563b774b4ef-otc-internal" pod "cd4f816b-930c-4909-b51a-d563b774b4ef" (UID: "cd4f816b-930c-4909-b51a-d563b774b4ef"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:10.969342 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.969315    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd4f816b-930c-4909-b51a-d563b774b4ef-monitoring-tls-certs" pod "cd4f816b-930c-4909-b51a-d563b774b4ef" (UID: "cd4f816b-930c-4909-b51a-d563b774b4ef"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:10.969457 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.969395    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4f816b-930c-4909-b51a-d563b774b4ef-kube-api-access-6g6tr" pod "cd4f816b-930c-4909-b51a-d563b774b4ef" (UID: "cd4f816b-930c-4909-b51a-d563b774b4ef"). InnerVolumeSpecName "kube-api-access-6g6tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:47:10.969457 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:10.969414    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd4f816b-930c-4909-b51a-d563b774b4ef-otlp-tls-certs" pod "cd4f816b-930c-4909-b51a-d563b774b4ef" (UID: "cd4f816b-930c-4909-b51a-d563b774b4ef"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:11.049871 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.049834    4960 generic.go:356] "Generic (PLEG): container finished" podID="cd4f816b-930c-4909-b51a-d563b774b4ef" containerID="15ec6bd293d086e075b28efb1653abe48548ee6fba6eab0b8b79385ddee73272" exitCode=0
Oct 30 03:47:11.050059 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.049888    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:11.050059 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.049919    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cd4f816b-930c-4909-b51a-d563b774b4ef","Type":"ContainerDied","Data":"15ec6bd293d086e075b28efb1653abe48548ee6fba6eab0b8b79385ddee73272"}
Oct 30 03:47:11.050059 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.050035    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cd4f816b-930c-4909-b51a-d563b774b4ef","Type":"ContainerDied","Data":"65934e00221d621e5f97b07b34d7e48bb3ad03cf76abbea8831a4d9d6700e3e5"}
Oct 30 03:47:11.050315 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.050072    4960 scope.go:122] "RemoveContainer" containerID="15ec6bd293d086e075b28efb1653abe48548ee6fba6eab0b8b79385ddee73272"
Oct 30 03:47:11.057119 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.057028    4960 generic.go:356] "Generic (PLEG): container finished" podID="5d494afe-0126-4460-9f0a-614f6eb4f0f1" containerID="9ae16178987a127988b4dd055b8b2c712e08b7c5c03379b9f682f6a30e3f3cbf" exitCode=0
Oct 30 03:47:11.057119 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.057096    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"
Oct 30 03:47:11.057307 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.057113    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g" event={"ID":"5d494afe-0126-4460-9f0a-614f6eb4f0f1","Type":"ContainerDied","Data":"9ae16178987a127988b4dd055b8b2c712e08b7c5c03379b9f682f6a30e3f3cbf"}
Oct 30 03:47:11.057307 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.057168    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g" event={"ID":"5d494afe-0126-4460-9f0a-614f6eb4f0f1","Type":"ContainerDied","Data":"ae410185b971afd9eac4d203d4b96eddcc0ecf78f278a83e8dac143f6acd9a53"}
Oct 30 03:47:11.067782 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.067257    4960 scope.go:122] "RemoveContainer" containerID="15ec6bd293d086e075b28efb1653abe48548ee6fba6eab0b8b79385ddee73272"
Oct 30 03:47:11.067782 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:47:11.067623    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15ec6bd293d086e075b28efb1653abe48548ee6fba6eab0b8b79385ddee73272\": container with ID starting with 15ec6bd293d086e075b28efb1653abe48548ee6fba6eab0b8b79385ddee73272 not found: ID does not exist" containerID="15ec6bd293d086e075b28efb1653abe48548ee6fba6eab0b8b79385ddee73272"
Oct 30 03:47:11.067782 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.067700    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15ec6bd293d086e075b28efb1653abe48548ee6fba6eab0b8b79385ddee73272"} err="failed to get container status \"15ec6bd293d086e075b28efb1653abe48548ee6fba6eab0b8b79385ddee73272\": rpc error: code = NotFound desc = could not find container \"15ec6bd293d086e075b28efb1653abe48548ee6fba6eab0b8b79385ddee73272\": container with ID starting with 15ec6bd293d086e075b28efb1653abe48548ee6fba6eab0b8b79385ddee73272 not found: ID does not exist"
Oct 30 03:47:11.067782 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.067724    4960 scope.go:122] "RemoveContainer" containerID="9ae16178987a127988b4dd055b8b2c712e08b7c5c03379b9f682f6a30e3f3cbf"
Oct 30 03:47:11.067782 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.067762    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g6tr\" (UniqueName: \"kubernetes.io/projected/cd4f816b-930c-4909-b51a-d563b774b4ef-kube-api-access-6g6tr\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:11.067782 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.067783    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4f816b-930c-4909-b51a-d563b774b4ef-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:11.068164 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.067798    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cd4f816b-930c-4909-b51a-d563b774b4ef-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:11.068164 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.067812    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4f816b-930c-4909-b51a-d563b774b4ef-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:11.082142 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.082122    4960 scope.go:122] "RemoveContainer" containerID="9ae16178987a127988b4dd055b8b2c712e08b7c5c03379b9f682f6a30e3f3cbf"
Oct 30 03:47:11.082460 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:47:11.082442    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae16178987a127988b4dd055b8b2c712e08b7c5c03379b9f682f6a30e3f3cbf\": container with ID starting with 9ae16178987a127988b4dd055b8b2c712e08b7c5c03379b9f682f6a30e3f3cbf not found: ID does not exist" containerID="9ae16178987a127988b4dd055b8b2c712e08b7c5c03379b9f682f6a30e3f3cbf"
Oct 30 03:47:11.082508 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.082468    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae16178987a127988b4dd055b8b2c712e08b7c5c03379b9f682f6a30e3f3cbf"} err="failed to get container status \"9ae16178987a127988b4dd055b8b2c712e08b7c5c03379b9f682f6a30e3f3cbf\": rpc error: code = NotFound desc = could not find container \"9ae16178987a127988b4dd055b8b2c712e08b7c5c03379b9f682f6a30e3f3cbf\": container with ID starting with 9ae16178987a127988b4dd055b8b2c712e08b7c5c03379b9f682f6a30e3f3cbf not found: ID does not exist"
Oct 30 03:47:11.135549 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.135519    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:47:11.137461 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.137433    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:47:11.153504 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.153471    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"]
Oct 30 03:47:11.154228 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.153749    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn" podUID="1524f4c0-dd9a-4c2b-811e-4e10af5a62fd" containerName="tempo" containerID="cri-o://2248957b195cc1db3244ded85e8f25bbb750fbc8357d0874e7e35f2102db1173" gracePeriod=30
Oct 30 03:47:11.155759 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.155686    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"]
Oct 30 03:47:11.156118 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.156088    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc" podUID="aa0b5af3-99b2-43d4-b875-d392051f7da8" containerName="tempo" containerID="cri-o://2edf6f154fe41047872a96fa4e505ef8f70f60495bf22485183d7c95e0954275" gracePeriod=30
Oct 30 03:47:11.163990 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.163938    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"]
Oct 30 03:47:11.164323 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.164194    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl" podUID="61bc4b06-9920-4266-a83e-59233486fc7b" containerName="tempo" containerID="cri-o://cff831dcf72d5e524d80ad1fd83c333a394d96be4c529047237f18f58fe0d670" gracePeriod=30
Oct 30 03:47:11.166355 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.166323    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"]
Oct 30 03:47:11.166785 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.166724    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf" podUID="d53f2afb-6b3d-4290-86f3-c663ef9734be" containerName="tempo" containerID="cri-o://3ceff0c4c9ac1a5141dffe6bf02eb1aab2cf22301c98e1668193c4b4eb50fda4" gracePeriod=30
Oct 30 03:47:11.168961 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.168911    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"]
Oct 30 03:47:11.173771 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.173522    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-4pv6g"]
Oct 30 03:47:11.426433 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.426404    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:47:11.504407 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.504330    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d494afe-0126-4460-9f0a-614f6eb4f0f1" path="/var/lib/kubelet/pods/5d494afe-0126-4460-9f0a-614f6eb4f0f1/volumes"
Oct 30 03:47:11.504786 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.504765    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd4f816b-930c-4909-b51a-d563b774b4ef" path="/var/lib/kubelet/pods/cd4f816b-930c-4909-b51a-d563b774b4ef/volumes"
Oct 30 03:47:11.571969 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.571940    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-tmp-storage\") pod \"aa0b5af3-99b2-43d4-b875-d392051f7da8\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") "
Oct 30 03:47:11.572109 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.571986    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-data-science-tempostack-ca-bundle\") pod \"aa0b5af3-99b2-43d4-b875-d392051f7da8\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") "
Oct 30 03:47:11.572109 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.572033    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-data-science-tempostack-distributor-mtls\") pod \"aa0b5af3-99b2-43d4-b875-d392051f7da8\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") "
Oct 30 03:47:11.572109 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.572054    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa0b5af3-99b2-43d4-b875-d392051f7da8-kube-api-access-brzk4\" (UniqueName: \"kubernetes.io/projected/aa0b5af3-99b2-43d4-b875-d392051f7da8-kube-api-access-brzk4\") pod \"aa0b5af3-99b2-43d4-b875-d392051f7da8\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") "
Oct 30 03:47:11.572315 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.572128    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-conf\") pod \"aa0b5af3-99b2-43d4-b875-d392051f7da8\" (UID: \"aa0b5af3-99b2-43d4-b875-d392051f7da8\") "
Oct 30 03:47:11.572315 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.572228    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-tmp-storage" pod "aa0b5af3-99b2-43d4-b875-d392051f7da8" (UID: "aa0b5af3-99b2-43d4-b875-d392051f7da8"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:47:11.572428 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.572365    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-tmp-storage\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:11.572485 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.572462    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-data-science-tempostack-ca-bundle" pod "aa0b5af3-99b2-43d4-b875-d392051f7da8" (UID: "aa0b5af3-99b2-43d4-b875-d392051f7da8"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:11.572564 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.572546    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-conf" pod "aa0b5af3-99b2-43d4-b875-d392051f7da8" (UID: "aa0b5af3-99b2-43d4-b875-d392051f7da8"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:11.574233 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.574162    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-data-science-tempostack-distributor-mtls" pod "aa0b5af3-99b2-43d4-b875-d392051f7da8" (UID: "aa0b5af3-99b2-43d4-b875-d392051f7da8"). InnerVolumeSpecName "tempo-data-science-tempostack-distributor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:11.574312 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.574234    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa0b5af3-99b2-43d4-b875-d392051f7da8-kube-api-access-brzk4" pod "aa0b5af3-99b2-43d4-b875-d392051f7da8" (UID: "aa0b5af3-99b2-43d4-b875-d392051f7da8"). InnerVolumeSpecName "kube-api-access-brzk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:47:11.673496 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.673472    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-data-science-tempostack-distributor-mtls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:11.673496 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.673495    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brzk4\" (UniqueName: \"kubernetes.io/projected/aa0b5af3-99b2-43d4-b875-d392051f7da8-kube-api-access-brzk4\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:11.673629 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.673506    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-conf\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:11.673629 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.673515    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa0b5af3-99b2-43d4-b875-d392051f7da8-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:11.921327 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:11.921302    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:12.064773 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.064744    4960 generic.go:356] "Generic (PLEG): container finished" podID="aa0b5af3-99b2-43d4-b875-d392051f7da8" containerID="2edf6f154fe41047872a96fa4e505ef8f70f60495bf22485183d7c95e0954275" exitCode=0
Oct 30 03:47:12.064889 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.064787    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"
Oct 30 03:47:12.064889 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.064815    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc" event={"ID":"aa0b5af3-99b2-43d4-b875-d392051f7da8","Type":"ContainerDied","Data":"2edf6f154fe41047872a96fa4e505ef8f70f60495bf22485183d7c95e0954275"}
Oct 30 03:47:12.064889 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.064857    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc" event={"ID":"aa0b5af3-99b2-43d4-b875-d392051f7da8","Type":"ContainerDied","Data":"1fd70eedc1b4b4df7ac50342fbabfd3fdab2eb349b7e892d620072961983f46e"}
Oct 30 03:47:12.064889 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.064880    4960 scope.go:122] "RemoveContainer" containerID="2edf6f154fe41047872a96fa4e505ef8f70f60495bf22485183d7c95e0954275"
Oct 30 03:47:12.073808 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.073787    4960 scope.go:122] "RemoveContainer" containerID="2edf6f154fe41047872a96fa4e505ef8f70f60495bf22485183d7c95e0954275"
Oct 30 03:47:12.074057 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:47:12.074040    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2edf6f154fe41047872a96fa4e505ef8f70f60495bf22485183d7c95e0954275\": container with ID starting with 2edf6f154fe41047872a96fa4e505ef8f70f60495bf22485183d7c95e0954275 not found: ID does not exist" containerID="2edf6f154fe41047872a96fa4e505ef8f70f60495bf22485183d7c95e0954275"
Oct 30 03:47:12.074120 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.074063    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2edf6f154fe41047872a96fa4e505ef8f70f60495bf22485183d7c95e0954275"} err="failed to get container status \"2edf6f154fe41047872a96fa4e505ef8f70f60495bf22485183d7c95e0954275\": rpc error: code = NotFound desc = could not find container \"2edf6f154fe41047872a96fa4e505ef8f70f60495bf22485183d7c95e0954275\": container with ID starting with 2edf6f154fe41047872a96fa4e505ef8f70f60495bf22485183d7c95e0954275 not found: ID does not exist"
Oct 30 03:47:12.122001 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.121974    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"]
Oct 30 03:47:12.135446 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.135421    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6cd5559486-gtqmc"]
Oct 30 03:47:12.619230 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.619204    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:47:12.783662 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.783599    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-tmp-storage\") pod \"61bc4b06-9920-4266-a83e-59233486fc7b\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") "
Oct 30 03:47:12.783662 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.783639    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61bc4b06-9920-4266-a83e-59233486fc7b-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/61bc4b06-9920-4266-a83e-59233486fc7b-storage-gcs-key\") pod \"61bc4b06-9920-4266-a83e-59233486fc7b\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") "
Oct 30 03:47:12.783827 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.783664    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-data-science-tempostack-ca-bundle\") pod \"61bc4b06-9920-4266-a83e-59233486fc7b\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") "
Oct 30 03:47:12.783827 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.783698    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61bc4b06-9920-4266-a83e-59233486fc7b-kube-api-access-l68z7\" (UniqueName: \"kubernetes.io/projected/61bc4b06-9920-4266-a83e-59233486fc7b-kube-api-access-l68z7\") pod \"61bc4b06-9920-4266-a83e-59233486fc7b\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") "
Oct 30 03:47:12.783827 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.783809    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-conf\") pod \"61bc4b06-9920-4266-a83e-59233486fc7b\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") "
Oct 30 03:47:12.783980 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.783859    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-data-science-tempostack-compactor-mtls\") pod \"61bc4b06-9920-4266-a83e-59233486fc7b\" (UID: \"61bc4b06-9920-4266-a83e-59233486fc7b\") "
Oct 30 03:47:12.783980 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.783947    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-tmp-storage" pod "61bc4b06-9920-4266-a83e-59233486fc7b" (UID: "61bc4b06-9920-4266-a83e-59233486fc7b"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:47:12.784229 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.784202    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-data-science-tempostack-ca-bundle" pod "61bc4b06-9920-4266-a83e-59233486fc7b" (UID: "61bc4b06-9920-4266-a83e-59233486fc7b"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:12.784229 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.784212    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-tmp-storage\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:12.784451 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.784251    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-conf" pod "61bc4b06-9920-4266-a83e-59233486fc7b" (UID: "61bc4b06-9920-4266-a83e-59233486fc7b"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:12.786035 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.786000    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61bc4b06-9920-4266-a83e-59233486fc7b-kube-api-access-l68z7" pod "61bc4b06-9920-4266-a83e-59233486fc7b" (UID: "61bc4b06-9920-4266-a83e-59233486fc7b"). InnerVolumeSpecName "kube-api-access-l68z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:47:12.786035 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.786009    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-data-science-tempostack-compactor-mtls" pod "61bc4b06-9920-4266-a83e-59233486fc7b" (UID: "61bc4b06-9920-4266-a83e-59233486fc7b"). InnerVolumeSpecName "tempo-data-science-tempostack-compactor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:12.786168 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.786040    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61bc4b06-9920-4266-a83e-59233486fc7b-storage-gcs-key" pod "61bc4b06-9920-4266-a83e-59233486fc7b" (UID: "61bc4b06-9920-4266-a83e-59233486fc7b"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:12.885483 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.885456    4960 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/61bc4b06-9920-4266-a83e-59233486fc7b-storage-gcs-key\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:12.885483 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.885476    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:12.885610 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.885491    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l68z7\" (UniqueName: \"kubernetes.io/projected/61bc4b06-9920-4266-a83e-59233486fc7b-kube-api-access-l68z7\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:12.885610 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.885501    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-conf\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:12.885610 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:12.885510    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/61bc4b06-9920-4266-a83e-59233486fc7b-tempo-data-science-tempostack-compactor-mtls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:13.070648 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:13.070624    4960 generic.go:356] "Generic (PLEG): container finished" podID="61bc4b06-9920-4266-a83e-59233486fc7b" containerID="cff831dcf72d5e524d80ad1fd83c333a394d96be4c529047237f18f58fe0d670" exitCode=0
Oct 30 03:47:13.070763 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:13.070678    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"
Oct 30 03:47:13.070763 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:13.070702    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl" event={"ID":"61bc4b06-9920-4266-a83e-59233486fc7b","Type":"ContainerDied","Data":"cff831dcf72d5e524d80ad1fd83c333a394d96be4c529047237f18f58fe0d670"}
Oct 30 03:47:13.070763 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:13.070750    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl" event={"ID":"61bc4b06-9920-4266-a83e-59233486fc7b","Type":"ContainerDied","Data":"12b37036922c5a9776ee87b3a109539a0df5bb9e145d59cad695a92ab24ac4a5"}
Oct 30 03:47:13.070885 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:13.070772    4960 scope.go:122] "RemoveContainer" containerID="cff831dcf72d5e524d80ad1fd83c333a394d96be4c529047237f18f58fe0d670"
Oct 30 03:47:13.079663 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:13.079639    4960 scope.go:122] "RemoveContainer" containerID="cff831dcf72d5e524d80ad1fd83c333a394d96be4c529047237f18f58fe0d670"
Oct 30 03:47:13.079892 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:47:13.079871    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff831dcf72d5e524d80ad1fd83c333a394d96be4c529047237f18f58fe0d670\": container with ID starting with cff831dcf72d5e524d80ad1fd83c333a394d96be4c529047237f18f58fe0d670 not found: ID does not exist" containerID="cff831dcf72d5e524d80ad1fd83c333a394d96be4c529047237f18f58fe0d670"
Oct 30 03:47:13.079945 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:13.079899    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff831dcf72d5e524d80ad1fd83c333a394d96be4c529047237f18f58fe0d670"} err="failed to get container status \"cff831dcf72d5e524d80ad1fd83c333a394d96be4c529047237f18f58fe0d670\": rpc error: code = NotFound desc = could not find container \"cff831dcf72d5e524d80ad1fd83c333a394d96be4c529047237f18f58fe0d670\": container with ID starting with cff831dcf72d5e524d80ad1fd83c333a394d96be4c529047237f18f58fe0d670 not found: ID does not exist"
Oct 30 03:47:13.124926 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:13.124901    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"]
Oct 30 03:47:13.138500 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:13.138480    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-59f4bb8c76-8z8hl"]
Oct 30 03:47:13.503502 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:13.503430    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61bc4b06-9920-4266-a83e-59233486fc7b" path="/var/lib/kubelet/pods/61bc4b06-9920-4266-a83e-59233486fc7b/volumes"
Oct 30 03:47:13.503827 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:13.503814    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa0b5af3-99b2-43d4-b875-d392051f7da8" path="/var/lib/kubelet/pods/aa0b5af3-99b2-43d4-b875-d392051f7da8/volumes"
Oct 30 03:47:13.564392 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:13.564365    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn" podUID="1524f4c0-dd9a-4c2b-811e-4e10af5a62fd" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:47:13.568264 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:13.568243    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf" podUID="d53f2afb-6b3d-4290-86f3-c663ef9734be" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:47:13.921635 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:13.921607    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:14.971575 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:14.971540    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:15.012332 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.012304    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"]
Oct 30 03:47:15.012630 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.012618    4960 cpu_manager.go:414] "containerMap: removing container" podUID="61bc4b06-9920-4266-a83e-59233486fc7b" containerName="tempo"
Oct 30 03:47:15.012676 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.012631    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61bc4b06-9920-4266-a83e-59233486fc7b" containerName="tempo"
Oct 30 03:47:15.012676 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.012648    4960 cpu_manager.go:414] "containerMap: removing container" podUID="cd4f816b-930c-4909-b51a-d563b774b4ef" containerName="otc-container"
Oct 30 03:47:15.012676 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.012653    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd4f816b-930c-4909-b51a-d563b774b4ef" containerName="otc-container"
Oct 30 03:47:15.012676 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.012666    4960 cpu_manager.go:414] "containerMap: removing container" podUID="5d494afe-0126-4460-9f0a-614f6eb4f0f1" containerName="korrel8r"
Oct 30 03:47:15.012676 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.012672    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d494afe-0126-4460-9f0a-614f6eb4f0f1" containerName="korrel8r"
Oct 30 03:47:15.012676 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.012677    4960 cpu_manager.go:414] "containerMap: removing container" podUID="aa0b5af3-99b2-43d4-b875-d392051f7da8" containerName="tempo"
Oct 30 03:47:15.012850 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.012683    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa0b5af3-99b2-43d4-b875-d392051f7da8" containerName="tempo"
Oct 30 03:47:15.012850 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.012739    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="61bc4b06-9920-4266-a83e-59233486fc7b" containerName="tempo"
Oct 30 03:47:15.012850 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.012749    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd4f816b-930c-4909-b51a-d563b774b4ef" containerName="otc-container"
Oct 30 03:47:15.012850 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.012755    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d494afe-0126-4460-9f0a-614f6eb4f0f1" containerName="korrel8r"
Oct 30 03:47:15.012850 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.012760    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa0b5af3-99b2-43d4-b875-d392051f7da8" containerName="tempo"
Oct 30 03:47:15.017130 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.017113    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"
Oct 30 03:47:15.034210 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.034165    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"]
Oct 30 03:47:15.046269 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.046250    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-pm7lh\""
Oct 30 03:47:15.046269 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.046260    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\""
Oct 30 03:47:15.046404 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.046364    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\""
Oct 30 03:47:15.081224 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.081200    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:15.203243 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.203221    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eda6feb4-b449-415d-a900-9c075f548b6b-config\") pod \"korrel8r-6644769df5-pwmcd\" (UID: \"eda6feb4-b449-415d-a900-9c075f548b6b\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"
Oct 30 03:47:15.203357 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.203253    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/eda6feb4-b449-415d-a900-9c075f548b6b-tls\") pod \"korrel8r-6644769df5-pwmcd\" (UID: \"eda6feb4-b449-415d-a900-9c075f548b6b\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"
Oct 30 03:47:15.203407 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.203384    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69qqq\" (UniqueName: \"kubernetes.io/projected/eda6feb4-b449-415d-a900-9c075f548b6b-kube-api-access-69qqq\") pod \"korrel8r-6644769df5-pwmcd\" (UID: \"eda6feb4-b449-415d-a900-9c075f548b6b\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"
Oct 30 03:47:15.212614 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.212586    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:15.303796 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.303734    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69qqq\" (UniqueName: \"kubernetes.io/projected/eda6feb4-b449-415d-a900-9c075f548b6b-kube-api-access-69qqq\") pod \"korrel8r-6644769df5-pwmcd\" (UID: \"eda6feb4-b449-415d-a900-9c075f548b6b\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"
Oct 30 03:47:15.303796 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.303768    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eda6feb4-b449-415d-a900-9c075f548b6b-config\") pod \"korrel8r-6644769df5-pwmcd\" (UID: \"eda6feb4-b449-415d-a900-9c075f548b6b\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"
Oct 30 03:47:15.303796 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.303792    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/eda6feb4-b449-415d-a900-9c075f548b6b-tls\") pod \"korrel8r-6644769df5-pwmcd\" (UID: \"eda6feb4-b449-415d-a900-9c075f548b6b\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"
Oct 30 03:47:15.304721 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.304650    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eda6feb4-b449-415d-a900-9c075f548b6b-config\") pod \"korrel8r-6644769df5-pwmcd\" (UID: \"eda6feb4-b449-415d-a900-9c075f548b6b\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"
Oct 30 03:47:15.306496 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.306475    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/eda6feb4-b449-415d-a900-9c075f548b6b-tls\") pod \"korrel8r-6644769df5-pwmcd\" (UID: \"eda6feb4-b449-415d-a900-9c075f548b6b\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"
Oct 30 03:47:15.321374 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.321346    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:15.324437 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.324412    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69qqq\" (UniqueName: \"kubernetes.io/projected/eda6feb4-b449-415d-a900-9c075f548b6b-kube-api-access-69qqq\") pod \"korrel8r-6644769df5-pwmcd\" (UID: \"eda6feb4-b449-415d-a900-9c075f548b6b\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"
Oct 30 03:47:15.329247 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.329201    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"
Oct 30 03:47:15.624476 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.624451    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"]
Oct 30 03:47:15.627093 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:47:15.627061    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda6feb4_b449_415d_a900_9c075f548b6b.slice/crio-3e78c666644ee85d7e1a0cb096d3805a7b74079b4d2c21489875c939445c51c9 WatchSource:0}: Error finding container 3e78c666644ee85d7e1a0cb096d3805a7b74079b4d2c21489875c939445c51c9: Status 404 returned error can't find the container with id 3e78c666644ee85d7e1a0cb096d3805a7b74079b4d2c21489875c939445c51c9
Oct 30 03:47:15.647480 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.647449    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:47:15.652695 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.652674    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:15.659228 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.659169    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:15.670547 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.670523    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:47:15.743030 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.742997    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:47:15.743158 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.743098    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-ptc4w\""
Oct 30 03:47:15.743158 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.743153    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\""
Oct 30 03:47:15.743292 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.743215    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:47:15.776357 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.776320    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:15.807214 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.807186    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjnsk\" (UniqueName: \"kubernetes.io/projected/7f267da4-679d-4f0b-a7cc-4add9b174f0e-kube-api-access-wjnsk\") pod \"data-science-collector-collector-1\" (UID: \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:15.807350 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.807267    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f267da4-679d-4f0b-a7cc-4add9b174f0e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:15.807350 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.807309    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f267da4-679d-4f0b-a7cc-4add9b174f0e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:15.807350 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.807342    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f267da4-679d-4f0b-a7cc-4add9b174f0e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:15.884262 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.884196    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:15.908423 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.908392    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f267da4-679d-4f0b-a7cc-4add9b174f0e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:15.908545 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.908485    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjnsk\" (UniqueName: \"kubernetes.io/projected/7f267da4-679d-4f0b-a7cc-4add9b174f0e-kube-api-access-wjnsk\") pod \"data-science-collector-collector-1\" (UID: \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:15.908545 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.908517    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f267da4-679d-4f0b-a7cc-4add9b174f0e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:15.908665 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.908557    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f267da4-679d-4f0b-a7cc-4add9b174f0e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:15.909223 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.909200    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f267da4-679d-4f0b-a7cc-4add9b174f0e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:15.911312 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.911287    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f267da4-679d-4f0b-a7cc-4add9b174f0e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:15.912344 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.912325    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f267da4-679d-4f0b-a7cc-4add9b174f0e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:15.929298 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.929275    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjnsk\" (UniqueName: \"kubernetes.io/projected/7f267da4-679d-4f0b-a7cc-4add9b174f0e-kube-api-access-wjnsk\") pod \"data-science-collector-collector-1\" (UID: \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:15.980236 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:15.980212    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:16.095104 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:16.095072    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd" event={"ID":"eda6feb4-b449-415d-a900-9c075f548b6b","Type":"ContainerStarted","Data":"ab82c80116634ae151e1d9c6c40c0c4ce21efe54d5bee0cb114b04843db10768"}
Oct 30 03:47:16.095259 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:16.095124    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd" event={"ID":"eda6feb4-b449-415d-a900-9c075f548b6b","Type":"ContainerStarted","Data":"3e78c666644ee85d7e1a0cb096d3805a7b74079b4d2c21489875c939445c51c9"}
Oct 30 03:47:16.095259 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:16.095170    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"
Oct 30 03:47:16.136209 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:16.136098    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd" podStartSLOduration=2.136085772 podStartE2EDuration="2.136085772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:16.134608143 +0000 UTC m=+3916.783656563" watchObservedRunningTime="2026-10-30 03:47:16.136085772 +0000 UTC m=+3916.785134190"
Oct 30 03:47:16.272560 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:16.272535    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:47:16.274491 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:47:16.274463    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f267da4_679d_4f0b_a7cc_4add9b174f0e.slice/crio-45cb987bff790bcf972ae5276f7abaed756026bdcb7e0bcded1b87d0fc5d4e69 WatchSource:0}: Error finding container 45cb987bff790bcf972ae5276f7abaed756026bdcb7e0bcded1b87d0fc5d4e69: Status 404 returned error can't find the container with id 45cb987bff790bcf972ae5276f7abaed756026bdcb7e0bcded1b87d0fc5d4e69
Oct 30 03:47:16.276592 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:16.276564    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:47:16.276689 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:16.276617    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:47:17.101399 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:17.101356    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7f267da4-679d-4f0b-a7cc-4add9b174f0e","Type":"ContainerStarted","Data":"7798e09b196d5ea8920a0ed276e61325473a45700b128a7b27a9876b7407102e"}
Oct 30 03:47:17.101782 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:17.101426    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7f267da4-679d-4f0b-a7cc-4add9b174f0e","Type":"ContainerStarted","Data":"45cb987bff790bcf972ae5276f7abaed756026bdcb7e0bcded1b87d0fc5d4e69"}
Oct 30 03:47:17.148422 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:17.148369    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.148357066 podStartE2EDuration="2.148357066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:17.147343095 +0000 UTC m=+3917.796391517" watchObservedRunningTime="2026-10-30 03:47:17.148357066 +0000 UTC m=+3917.797405531"
Oct 30 03:47:17.311809 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:17.311773    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:17.447706 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:17.447610    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:23.564911 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:23.564857    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn" podUID="1524f4c0-dd9a-4c2b-811e-4e10af5a62fd" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:47:23.595593 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:23.595553    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf" podUID="d53f2afb-6b3d-4290-86f3-c663ef9734be" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:47:23.953497 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:47:23.953462    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:47:24.234022 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:24.233887    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:24.340025 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:24.339995    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:27.104119 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:27.104080    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"
Oct 30 03:47:31.904519 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:31.904451    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:32.913123 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:32.913085    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"]
Oct 30 03:47:32.913539 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:32.913350    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd" podUID="eda6feb4-b449-415d-a900-9c075f548b6b" containerName="korrel8r" containerID="cri-o://ab82c80116634ae151e1d9c6c40c0c4ce21efe54d5bee0cb114b04843db10768" gracePeriod=30
Oct 30 03:47:33.163522 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.163470    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"
Oct 30 03:47:33.178219 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.178195    4960 generic.go:356] "Generic (PLEG): container finished" podID="eda6feb4-b449-415d-a900-9c075f548b6b" containerID="ab82c80116634ae151e1d9c6c40c0c4ce21efe54d5bee0cb114b04843db10768" exitCode=0
Oct 30 03:47:33.178312 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.178233    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"
Oct 30 03:47:33.178312 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.178239    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd" event={"ID":"eda6feb4-b449-415d-a900-9c075f548b6b","Type":"ContainerDied","Data":"ab82c80116634ae151e1d9c6c40c0c4ce21efe54d5bee0cb114b04843db10768"}
Oct 30 03:47:33.178312 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.178282    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-pwmcd" event={"ID":"eda6feb4-b449-415d-a900-9c075f548b6b","Type":"ContainerDied","Data":"3e78c666644ee85d7e1a0cb096d3805a7b74079b4d2c21489875c939445c51c9"}
Oct 30 03:47:33.178312 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.178305    4960 scope.go:122] "RemoveContainer" containerID="ab82c80116634ae151e1d9c6c40c0c4ce21efe54d5bee0cb114b04843db10768"
Oct 30 03:47:33.186382 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.186358    4960 scope.go:122] "RemoveContainer" containerID="ab82c80116634ae151e1d9c6c40c0c4ce21efe54d5bee0cb114b04843db10768"
Oct 30 03:47:33.186638 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:47:33.186617    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab82c80116634ae151e1d9c6c40c0c4ce21efe54d5bee0cb114b04843db10768\": container with ID starting with ab82c80116634ae151e1d9c6c40c0c4ce21efe54d5bee0cb114b04843db10768 not found: ID does not exist" containerID="ab82c80116634ae151e1d9c6c40c0c4ce21efe54d5bee0cb114b04843db10768"
Oct 30 03:47:33.186684 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.186653    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab82c80116634ae151e1d9c6c40c0c4ce21efe54d5bee0cb114b04843db10768"} err="failed to get container status \"ab82c80116634ae151e1d9c6c40c0c4ce21efe54d5bee0cb114b04843db10768\": rpc error: code = NotFound desc = could not find container \"ab82c80116634ae151e1d9c6c40c0c4ce21efe54d5bee0cb114b04843db10768\": container with ID starting with ab82c80116634ae151e1d9c6c40c0c4ce21efe54d5bee0cb114b04843db10768 not found: ID does not exist"
Oct 30 03:47:33.228044 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.228019    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eda6feb4-b449-415d-a900-9c075f548b6b-tls\" (UniqueName: \"kubernetes.io/secret/eda6feb4-b449-415d-a900-9c075f548b6b-tls\") pod \"eda6feb4-b449-415d-a900-9c075f548b6b\" (UID: \"eda6feb4-b449-415d-a900-9c075f548b6b\") "
Oct 30 03:47:33.228211 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.228064    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eda6feb4-b449-415d-a900-9c075f548b6b-kube-api-access-69qqq\" (UniqueName: \"kubernetes.io/projected/eda6feb4-b449-415d-a900-9c075f548b6b-kube-api-access-69qqq\") pod \"eda6feb4-b449-415d-a900-9c075f548b6b\" (UID: \"eda6feb4-b449-415d-a900-9c075f548b6b\") "
Oct 30 03:47:33.228211 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.228111    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eda6feb4-b449-415d-a900-9c075f548b6b-config\" (UniqueName: \"kubernetes.io/configmap/eda6feb4-b449-415d-a900-9c075f548b6b-config\") pod \"eda6feb4-b449-415d-a900-9c075f548b6b\" (UID: \"eda6feb4-b449-415d-a900-9c075f548b6b\") "
Oct 30 03:47:33.228544 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.228519    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda6feb4-b449-415d-a900-9c075f548b6b-config" pod "eda6feb4-b449-415d-a900-9c075f548b6b" (UID: "eda6feb4-b449-415d-a900-9c075f548b6b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:33.230338 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.230317    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda6feb4-b449-415d-a900-9c075f548b6b-kube-api-access-69qqq" pod "eda6feb4-b449-415d-a900-9c075f548b6b" (UID: "eda6feb4-b449-415d-a900-9c075f548b6b"). InnerVolumeSpecName "kube-api-access-69qqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:47:33.230412 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.230354    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eda6feb4-b449-415d-a900-9c075f548b6b-tls" pod "eda6feb4-b449-415d-a900-9c075f548b6b" (UID: "eda6feb4-b449-415d-a900-9c075f548b6b"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:33.326581 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.326550    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:47:33.326842 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.326787    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="7f267da4-679d-4f0b-a7cc-4add9b174f0e" containerName="otc-container" containerID="cri-o://7798e09b196d5ea8920a0ed276e61325473a45700b128a7b27a9876b7407102e" gracePeriod=30
Oct 30 03:47:33.329208 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.329162    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69qqq\" (UniqueName: \"kubernetes.io/projected/eda6feb4-b449-415d-a900-9c075f548b6b-kube-api-access-69qqq\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:33.329314 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.329214    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eda6feb4-b449-415d-a900-9c075f548b6b-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:33.329314 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.329232    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/eda6feb4-b449-415d-a900-9c075f548b6b-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:33.387696 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.387666    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:33.504119 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.502266    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:33.529005 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.528976    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"]
Oct 30 03:47:33.536851 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.536824    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-pwmcd"]
Oct 30 03:47:33.564709 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.564681    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn" podUID="1524f4c0-dd9a-4c2b-811e-4e10af5a62fd" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:47:33.564847 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.564775    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:47:33.587238 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.587172    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:33.596672 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.596649    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf" podUID="d53f2afb-6b3d-4290-86f3-c663ef9734be" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:47:33.596782 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.596748    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:47:33.632389 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.632368    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f267da4-679d-4f0b-a7cc-4add9b174f0e-kube-api-access-wjnsk\" (UniqueName: \"kubernetes.io/projected/7f267da4-679d-4f0b-a7cc-4add9b174f0e-kube-api-access-wjnsk\") pod \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\" (UID: \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\") "
Oct 30 03:47:33.632493 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.632401    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f267da4-679d-4f0b-a7cc-4add9b174f0e-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f267da4-679d-4f0b-a7cc-4add9b174f0e-monitoring-tls-certs\") pod \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\" (UID: \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\") "
Oct 30 03:47:33.632493 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.632449    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f267da4-679d-4f0b-a7cc-4add9b174f0e-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f267da4-679d-4f0b-a7cc-4add9b174f0e-otlp-tls-certs\") pod \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\" (UID: \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\") "
Oct 30 03:47:33.632493 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.632471    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f267da4-679d-4f0b-a7cc-4add9b174f0e-otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f267da4-679d-4f0b-a7cc-4add9b174f0e-otc-internal\") pod \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\" (UID: \"7f267da4-679d-4f0b-a7cc-4add9b174f0e\") "
Oct 30 03:47:33.632869 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.632847    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f267da4-679d-4f0b-a7cc-4add9b174f0e-otc-internal" pod "7f267da4-679d-4f0b-a7cc-4add9b174f0e" (UID: "7f267da4-679d-4f0b-a7cc-4add9b174f0e"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:33.634605 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.634585    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f267da4-679d-4f0b-a7cc-4add9b174f0e-otlp-tls-certs" pod "7f267da4-679d-4f0b-a7cc-4add9b174f0e" (UID: "7f267da4-679d-4f0b-a7cc-4add9b174f0e"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:33.635234 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.635219    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f267da4-679d-4f0b-a7cc-4add9b174f0e-monitoring-tls-certs" pod "7f267da4-679d-4f0b-a7cc-4add9b174f0e" (UID: "7f267da4-679d-4f0b-a7cc-4add9b174f0e"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:33.635315 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.635252    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f267da4-679d-4f0b-a7cc-4add9b174f0e-kube-api-access-wjnsk" pod "7f267da4-679d-4f0b-a7cc-4add9b174f0e" (UID: "7f267da4-679d-4f0b-a7cc-4add9b174f0e"). InnerVolumeSpecName "kube-api-access-wjnsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:47:33.639559 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.639540    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:33.733954 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.733889    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjnsk\" (UniqueName: \"kubernetes.io/projected/7f267da4-679d-4f0b-a7cc-4add9b174f0e-kube-api-access-wjnsk\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:33.733954 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.733914    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f267da4-679d-4f0b-a7cc-4add9b174f0e-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:33.733954 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.733923    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f267da4-679d-4f0b-a7cc-4add9b174f0e-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:33.733954 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.733932    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f267da4-679d-4f0b-a7cc-4add9b174f0e-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:33.851021 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.850984    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:33.979694 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:33.979663    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:34.185540 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:34.185509    4960 generic.go:356] "Generic (PLEG): container finished" podID="7f267da4-679d-4f0b-a7cc-4add9b174f0e" containerID="7798e09b196d5ea8920a0ed276e61325473a45700b128a7b27a9876b7407102e" exitCode=0
Oct 30 03:47:34.185712 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:34.185552    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7f267da4-679d-4f0b-a7cc-4add9b174f0e","Type":"ContainerDied","Data":"7798e09b196d5ea8920a0ed276e61325473a45700b128a7b27a9876b7407102e"}
Oct 30 03:47:34.185712 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:34.185585    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:34.185712 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:34.185633    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7f267da4-679d-4f0b-a7cc-4add9b174f0e","Type":"ContainerDied","Data":"45cb987bff790bcf972ae5276f7abaed756026bdcb7e0bcded1b87d0fc5d4e69"}
Oct 30 03:47:34.185712 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:34.185664    4960 scope.go:122] "RemoveContainer" containerID="7798e09b196d5ea8920a0ed276e61325473a45700b128a7b27a9876b7407102e"
Oct 30 03:47:34.194221 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:34.194198    4960 scope.go:122] "RemoveContainer" containerID="7798e09b196d5ea8920a0ed276e61325473a45700b128a7b27a9876b7407102e"
Oct 30 03:47:34.194467 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:47:34.194450    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7798e09b196d5ea8920a0ed276e61325473a45700b128a7b27a9876b7407102e\": container with ID starting with 7798e09b196d5ea8920a0ed276e61325473a45700b128a7b27a9876b7407102e not found: ID does not exist" containerID="7798e09b196d5ea8920a0ed276e61325473a45700b128a7b27a9876b7407102e"
Oct 30 03:47:34.194517 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:34.194475    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7798e09b196d5ea8920a0ed276e61325473a45700b128a7b27a9876b7407102e"} err="failed to get container status \"7798e09b196d5ea8920a0ed276e61325473a45700b128a7b27a9876b7407102e\": rpc error: code = NotFound desc = could not find container \"7798e09b196d5ea8920a0ed276e61325473a45700b128a7b27a9876b7407102e\": container with ID starting with 7798e09b196d5ea8920a0ed276e61325473a45700b128a7b27a9876b7407102e not found: ID does not exist"
Oct 30 03:47:34.240209 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:34.240189    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:47:34.243961 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:34.243942    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:47:35.502662 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:35.502633    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f267da4-679d-4f0b-a7cc-4add9b174f0e" path="/var/lib/kubelet/pods/7f267da4-679d-4f0b-a7cc-4add9b174f0e/volumes"
Oct 30 03:47:35.503031 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:35.503014    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eda6feb4-b449-415d-a900-9c075f548b6b" path="/var/lib/kubelet/pods/eda6feb4-b449-415d-a900-9c075f548b6b/volumes"
Oct 30 03:47:37.425926 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.425891    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:37.561830 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.561800    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:37.701060 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.700982    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:37.839330 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.839295    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"]
Oct 30 03:47:37.839690 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.839673    4960 cpu_manager.go:414] "containerMap: removing container" podUID="7f267da4-679d-4f0b-a7cc-4add9b174f0e" containerName="otc-container"
Oct 30 03:47:37.839777 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.839693    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f267da4-679d-4f0b-a7cc-4add9b174f0e" containerName="otc-container"
Oct 30 03:47:37.839777 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.839716    4960 cpu_manager.go:414] "containerMap: removing container" podUID="eda6feb4-b449-415d-a900-9c075f548b6b" containerName="korrel8r"
Oct 30 03:47:37.839777 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.839723    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eda6feb4-b449-415d-a900-9c075f548b6b" containerName="korrel8r"
Oct 30 03:47:37.839926 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.839846    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f267da4-679d-4f0b-a7cc-4add9b174f0e" containerName="otc-container"
Oct 30 03:47:37.839926 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.839862    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="eda6feb4-b449-415d-a900-9c075f548b6b" containerName="korrel8r"
Oct 30 03:47:37.842870 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.842849    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"
Oct 30 03:47:37.851504 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.851479    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:37.862292 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.862270    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"]
Oct 30 03:47:37.866323 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.866300    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-422mq\" (UniqueName: \"kubernetes.io/projected/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-kube-api-access-422mq\") pod \"korrel8r-5cbcc89b8f-mdwwk\" (UID: \"6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"
Oct 30 03:47:37.866414 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.866369    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-tls\") pod \"korrel8r-5cbcc89b8f-mdwwk\" (UID: \"6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"
Oct 30 03:47:37.866414 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.866402    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-config\") pod \"korrel8r-5cbcc89b8f-mdwwk\" (UID: \"6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"
Oct 30 03:47:37.931737 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.931712    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\""
Oct 30 03:47:37.931737 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.931727    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-zdpjk\""
Oct 30 03:47:37.931844 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.931720    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\""
Oct 30 03:47:37.963123 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.963069    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:37.967834 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.967814    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-422mq\" (UniqueName: \"kubernetes.io/projected/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-kube-api-access-422mq\") pod \"korrel8r-5cbcc89b8f-mdwwk\" (UID: \"6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"
Oct 30 03:47:37.967916 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.967855    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-tls\") pod \"korrel8r-5cbcc89b8f-mdwwk\" (UID: \"6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"
Oct 30 03:47:37.967916 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.967879    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-config\") pod \"korrel8r-5cbcc89b8f-mdwwk\" (UID: \"6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"
Oct 30 03:47:37.968663 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.968644    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-config\") pod \"korrel8r-5cbcc89b8f-mdwwk\" (UID: \"6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"
Oct 30 03:47:37.970309 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.970289    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-tls\") pod \"korrel8r-5cbcc89b8f-mdwwk\" (UID: \"6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"
Oct 30 03:47:37.989497 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:37.989477    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-422mq\" (UniqueName: \"kubernetes.io/projected/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-kube-api-access-422mq\") pod \"korrel8r-5cbcc89b8f-mdwwk\" (UID: \"6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"
Oct 30 03:47:38.088474 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:38.088451    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:38.155446 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:38.155423    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"
Oct 30 03:47:38.290276 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:38.290201    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:38.417431 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:38.417401    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:38.448345 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:38.448323    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"]
Oct 30 03:47:38.450377 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:47:38.450343    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2eaaaa_10f3_4545_8865_74ecb9ff4bfb.slice/crio-15298efc949cbddd867a6fedff9e8747a7ee00001b9cc3da8084b78c6a0792f8 WatchSource:0}: Error finding container 15298efc949cbddd867a6fedff9e8747a7ee00001b9cc3da8084b78c6a0792f8: Status 404 returned error can't find the container with id 15298efc949cbddd867a6fedff9e8747a7ee00001b9cc3da8084b78c6a0792f8
Oct 30 03:47:38.669808 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:38.669779    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:39.212099 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.212069    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk" event={"ID":"6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb","Type":"ContainerStarted","Data":"29e404c7f2875c58d8d2c346a092aba0c06abab63dfbd37ed1eb55d12a3c2c59"}
Oct 30 03:47:39.212268 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.212123    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk" event={"ID":"6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb","Type":"ContainerStarted","Data":"15298efc949cbddd867a6fedff9e8747a7ee00001b9cc3da8084b78c6a0792f8"}
Oct 30 03:47:39.212268 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.212162    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"
Oct 30 03:47:39.327361 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.327305    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk" podStartSLOduration=2.327286446 podStartE2EDuration="2.327286446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:39.26185399 +0000 UTC m=+3939.910902413" watchObservedRunningTime="2026-10-30 03:47:39.327286446 +0000 UTC m=+3939.976334870"
Oct 30 03:47:39.328223 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.328164    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:47:39.332339 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.332320    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.338976 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.338952    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:39.356143 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.356118    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:47:39.368398 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.368378    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wtzv9\""
Oct 30 03:47:39.368497 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.368438    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:47:39.368497 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.368463    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:47:39.368580 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.368521    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:47:39.368580 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.368543    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:47:39.381046 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.381019    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a01e9f19-e49e-4387-a8d2-d651a6120590-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.381140 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.381052    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.381217 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.381160    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwbtb\" (UniqueName: \"kubernetes.io/projected/a01e9f19-e49e-4387-a8d2-d651a6120590-kube-api-access-fwbtb\") pod \"data-science-collector-collector-1\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.381254 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.381218    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.381300 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.381282    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.381335 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.381315    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.418676 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.418641    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:47:39.476284 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.476200    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:39.482391 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.482359    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a01e9f19-e49e-4387-a8d2-d651a6120590-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.482510 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.482401    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.482510 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.482464    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwbtb\" (UniqueName: \"kubernetes.io/projected/a01e9f19-e49e-4387-a8d2-d651a6120590-kube-api-access-fwbtb\") pod \"data-science-collector-collector-1\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.482510 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.482483    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.482653 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.482512    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.482653 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.482528    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.483417 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.483273    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a01e9f19-e49e-4387-a8d2-d651a6120590-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.485089 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.485059    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.485201 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.485093    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.485201 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.485120    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.485201 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.485135    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.503673 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.503650    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwbtb\" (UniqueName: \"kubernetes.io/projected/a01e9f19-e49e-4387-a8d2-d651a6120590-kube-api-access-fwbtb\") pod \"data-science-collector-collector-1\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.592810 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.592779    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:39.646699 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.646677    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:47:39.940983 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.940961    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:47:39.942134 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:47:39.942100    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01e9f19_e49e_4387_a8d2_d651a6120590.slice/crio-9acd7a085c85791e9346a8830eb92ce8d286a5a87f3df10f93f62b4bb0239194 WatchSource:0}: Error finding container 9acd7a085c85791e9346a8830eb92ce8d286a5a87f3df10f93f62b4bb0239194: Status 404 returned error can't find the container with id 9acd7a085c85791e9346a8830eb92ce8d286a5a87f3df10f93f62b4bb0239194
Oct 30 03:47:39.944286 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.944254    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:47:39.944351 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:39.944316    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:47:40.219302 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:40.219223    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a01e9f19-e49e-4387-a8d2-d651a6120590","Type":"ContainerStarted","Data":"ce16dfbe7d4807b5dbbfa284861b669b8b4dc94505ed329eed2e4a8b4c99f4dc"}
Oct 30 03:47:40.219302 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:40.219284    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a01e9f19-e49e-4387-a8d2-d651a6120590","Type":"ContainerStarted","Data":"9acd7a085c85791e9346a8830eb92ce8d286a5a87f3df10f93f62b4bb0239194"}
Oct 30 03:47:40.263927 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:40.263881    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.263867654 podStartE2EDuration="1.263867654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:40.262407916 +0000 UTC m=+3940.911456362" watchObservedRunningTime="2026-10-30 03:47:40.263867654 +0000 UTC m=+3940.912916155"
Oct 30 03:47:41.227542 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.227514    4960 generic.go:356] "Generic (PLEG): container finished" podID="d53f2afb-6b3d-4290-86f3-c663ef9734be" containerID="3ceff0c4c9ac1a5141dffe6bf02eb1aab2cf22301c98e1668193c4b4eb50fda4" exitCode=137
Oct 30 03:47:41.227899 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.227610    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf" event={"ID":"d53f2afb-6b3d-4290-86f3-c663ef9734be","Type":"ContainerDied","Data":"3ceff0c4c9ac1a5141dffe6bf02eb1aab2cf22301c98e1668193c4b4eb50fda4"}
Oct 30 03:47:41.229561 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.229531    4960 generic.go:356] "Generic (PLEG): container finished" podID="1524f4c0-dd9a-4c2b-811e-4e10af5a62fd" containerID="2248957b195cc1db3244ded85e8f25bbb750fbc8357d0874e7e35f2102db1173" exitCode=137
Oct 30 03:47:41.229944 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.229921    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn" event={"ID":"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd","Type":"ContainerDied","Data":"2248957b195cc1db3244ded85e8f25bbb750fbc8357d0874e7e35f2102db1173"}
Oct 30 03:47:41.325761 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.325737    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:47:41.366038 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.366010    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:47:41.400229 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.400156    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-data-science-tempostack-query-frontend-mtls\") pod \"d53f2afb-6b3d-4290-86f3-c663ef9734be\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") "
Oct 30 03:47:41.400364 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.400242    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-data-science-tempostack-ca-bundle\") pod \"d53f2afb-6b3d-4290-86f3-c663ef9734be\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") "
Oct 30 03:47:41.400364 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.400293    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-storage-gcs-key\") pod \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") "
Oct 30 03:47:41.400364 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.400341    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d53f2afb-6b3d-4290-86f3-c663ef9734be-kube-api-access-5s8nm\" (UniqueName: \"kubernetes.io/projected/d53f2afb-6b3d-4290-86f3-c663ef9734be-kube-api-access-5s8nm\") pod \"d53f2afb-6b3d-4290-86f3-c663ef9734be\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") "
Oct 30 03:47:41.400502 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.400385    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-tmp-storage\") pod \"d53f2afb-6b3d-4290-86f3-c663ef9734be\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") "
Oct 30 03:47:41.400502 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.400420    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d53f2afb-6b3d-4290-86f3-c663ef9734be-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d53f2afb-6b3d-4290-86f3-c663ef9734be-storage-gcs-key\") pod \"d53f2afb-6b3d-4290-86f3-c663ef9734be\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") "
Oct 30 03:47:41.400502 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.400451    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-data-science-tempostack-ca-bundle\") pod \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") "
Oct 30 03:47:41.400502 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.400481    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-conf\") pod \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") "
Oct 30 03:47:41.400776 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.400747    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-data-science-tempostack-ca-bundle" pod "d53f2afb-6b3d-4290-86f3-c663ef9734be" (UID: "d53f2afb-6b3d-4290-86f3-c663ef9734be"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:41.400842 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.400787    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-tmp-storage" pod "d53f2afb-6b3d-4290-86f3-c663ef9734be" (UID: "d53f2afb-6b3d-4290-86f3-c663ef9734be"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:47:41.400893 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.400848    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-data-science-tempostack-ca-bundle" pod "1524f4c0-dd9a-4c2b-811e-4e10af5a62fd" (UID: "1524f4c0-dd9a-4c2b-811e-4e10af5a62fd"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:41.401092 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.401007    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-tmp-storage\") pod \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") "
Oct 30 03:47:41.401813 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.401231    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-data-science-tempostack-querier-mtls\") pod \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") "
Oct 30 03:47:41.401813 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.401285    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-conf\") pod \"d53f2afb-6b3d-4290-86f3-c663ef9734be\" (UID: \"d53f2afb-6b3d-4290-86f3-c663ef9734be\") "
Oct 30 03:47:41.401813 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.401339    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-kube-api-access-p2glp\" (UniqueName: \"kubernetes.io/projected/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-kube-api-access-p2glp\") pod \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\" (UID: \"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd\") "
Oct 30 03:47:41.401813 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.401357    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-tmp-storage" pod "1524f4c0-dd9a-4c2b-811e-4e10af5a62fd" (UID: "1524f4c0-dd9a-4c2b-811e-4e10af5a62fd"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:47:41.401813 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.401364    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-conf" pod "1524f4c0-dd9a-4c2b-811e-4e10af5a62fd" (UID: "1524f4c0-dd9a-4c2b-811e-4e10af5a62fd"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:41.401813 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.401585    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-tmp-storage\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:41.401813 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.401605    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:41.401813 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.401622    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-conf\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:41.401813 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.401638    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-tmp-storage\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:41.401813 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.401653    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:41.402437 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.401773    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-conf" pod "d53f2afb-6b3d-4290-86f3-c663ef9734be" (UID: "d53f2afb-6b3d-4290-86f3-c663ef9734be"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:41.402937 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.402906    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-data-science-tempostack-query-frontend-mtls" pod "d53f2afb-6b3d-4290-86f3-c663ef9734be" (UID: "d53f2afb-6b3d-4290-86f3-c663ef9734be"). InnerVolumeSpecName "tempo-data-science-tempostack-query-frontend-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:41.403141 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.403120    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d53f2afb-6b3d-4290-86f3-c663ef9734be-storage-gcs-key" pod "d53f2afb-6b3d-4290-86f3-c663ef9734be" (UID: "d53f2afb-6b3d-4290-86f3-c663ef9734be"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:41.403237 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.403125    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d53f2afb-6b3d-4290-86f3-c663ef9734be-kube-api-access-5s8nm" pod "d53f2afb-6b3d-4290-86f3-c663ef9734be" (UID: "d53f2afb-6b3d-4290-86f3-c663ef9734be"). InnerVolumeSpecName "kube-api-access-5s8nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:47:41.403237 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.403228    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-storage-gcs-key" pod "1524f4c0-dd9a-4c2b-811e-4e10af5a62fd" (UID: "1524f4c0-dd9a-4c2b-811e-4e10af5a62fd"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:41.403494 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.403457    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-data-science-tempostack-querier-mtls" pod "1524f4c0-dd9a-4c2b-811e-4e10af5a62fd" (UID: "1524f4c0-dd9a-4c2b-811e-4e10af5a62fd"). InnerVolumeSpecName "tempo-data-science-tempostack-querier-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:41.403736 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.403719    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-kube-api-access-p2glp" pod "1524f4c0-dd9a-4c2b-811e-4e10af5a62fd" (UID: "1524f4c0-dd9a-4c2b-811e-4e10af5a62fd"). InnerVolumeSpecName "kube-api-access-p2glp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:47:41.502137 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.502045    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s8nm\" (UniqueName: \"kubernetes.io/projected/d53f2afb-6b3d-4290-86f3-c663ef9734be-kube-api-access-5s8nm\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:41.502137 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.502072    4960 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d53f2afb-6b3d-4290-86f3-c663ef9734be-storage-gcs-key\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:41.502137 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.502083    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-tempo-data-science-tempostack-querier-mtls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:41.502137 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.502096    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-conf\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:41.502137 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.502106    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2glp\" (UniqueName: \"kubernetes.io/projected/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-kube-api-access-p2glp\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:41.502137 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.502119    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/d53f2afb-6b3d-4290-86f3-c663ef9734be-tempo-data-science-tempostack-query-frontend-mtls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:41.502137 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:41.502130    4960 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd-storage-gcs-key\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:47:42.224406 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.224372    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:42.235571 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.235546    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"
Oct 30 03:47:42.235992 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.235553    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf" event={"ID":"d53f2afb-6b3d-4290-86f3-c663ef9734be","Type":"ContainerDied","Data":"d85292650f5d63caee3c26a7f3679901e181e10478dfb4aeb589f6c77a995dc6"}
Oct 30 03:47:42.235992 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.235664    4960 scope.go:122] "RemoveContainer" containerID="3ceff0c4c9ac1a5141dffe6bf02eb1aab2cf22301c98e1668193c4b4eb50fda4"
Oct 30 03:47:42.237206 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.237168    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn" event={"ID":"1524f4c0-dd9a-4c2b-811e-4e10af5a62fd","Type":"ContainerDied","Data":"5c6a20d8b9875c741f14a4afcc2d98eb0990bf94c6caf41598e6fd9ad5bd1003"}
Oct 30 03:47:42.237291 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.237281    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"
Oct 30 03:47:42.244462 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.244439    4960 scope.go:122] "RemoveContainer" containerID="2248957b195cc1db3244ded85e8f25bbb750fbc8357d0874e7e35f2102db1173"
Oct 30 03:47:42.292973 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.292953    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"]
Oct 30 03:47:42.306709 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.306686    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-84f565bdd4-c7ksn"]
Oct 30 03:47:42.334457 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.334434    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"]
Oct 30 03:47:42.338001 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.337983    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-85bfb8b7bc-vdtkf"]
Oct 30 03:47:42.452074 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.452010    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:42.557916 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.557892    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:42.866417 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.866386    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:47:42.866784 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.866769    4960 cpu_manager.go:414] "containerMap: removing container" podUID="1524f4c0-dd9a-4c2b-811e-4e10af5a62fd" containerName="tempo"
Oct 30 03:47:42.866835 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.866786    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1524f4c0-dd9a-4c2b-811e-4e10af5a62fd" containerName="tempo"
Oct 30 03:47:42.866835 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.866795    4960 cpu_manager.go:414] "containerMap: removing container" podUID="d53f2afb-6b3d-4290-86f3-c663ef9734be" containerName="tempo"
Oct 30 03:47:42.866835 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.866799    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d53f2afb-6b3d-4290-86f3-c663ef9734be" containerName="tempo"
Oct 30 03:47:42.866928 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.866898    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="1524f4c0-dd9a-4c2b-811e-4e10af5a62fd" containerName="tempo"
Oct 30 03:47:42.866928 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.866911    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="d53f2afb-6b3d-4290-86f3-c663ef9734be" containerName="tempo"
Oct 30 03:47:42.871987 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.871964    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:42.891683 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.891653    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:42.894239 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.892428    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 03:47:42.894239 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.892363    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 03:47:42.894239 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.892659    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 03:47:42.894239 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.892974    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 03:47:42.903946 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.903919    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 03:47:42.903946 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.903933    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-5cs6w\""
Oct 30 03:47:42.903946 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.903919    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 03:47:42.904154 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.903983    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 03:47:42.904154 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.904060    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 03:47:42.904428 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.904407    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 03:47:42.906143 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.906118    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:47:42.907687 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.907667    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 03:47:42.911213 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.911195    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:42.937604 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.937567    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 03:47:42.990380 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.990357    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\""
Oct 30 03:47:42.990507 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.990378    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\""
Oct 30 03:47:42.990507 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.990402    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\""
Oct 30 03:47:42.990507 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.990425    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\""
Oct 30 03:47:42.990507 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:42.990462    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-fp6sj\""
Oct 30 03:47:43.011480 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.011446    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:43.013611 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.013591    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.013684 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.013624    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rprrp\" (UniqueName: \"kubernetes.io/projected/fb57c6c2-5791-478a-bfa2-c98fedc041ed-kube-api-access-rprrp\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.013684 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.013643    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb57c6c2-5791-478a-bfa2-c98fedc041ed-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.013684 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.013671    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.013818 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.013734    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.013867 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.013852    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.013904 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.013878    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.013904 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.013899    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.013967 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.013921    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.014025 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.014010    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.014062 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.014048    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb57c6c2-5791-478a-bfa2-c98fedc041ed-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.014108 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.014076    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.114425 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.114402    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.114532 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.114437    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33a822a2-33ea-4a2c-93e8-383364e7593d-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.114532 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.114464    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33a822a2-33ea-4a2c-93e8-383364e7593d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.114610 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.114554    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb57c6c2-5791-478a-bfa2-c98fedc041ed-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.114654 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.114605    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.114654 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.114634    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.114759 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.114663    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/33a822a2-33ea-4a2c-93e8-383364e7593d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.114759 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.114707    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.114873 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.114781    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rprrp\" (UniqueName: \"kubernetes.io/projected/fb57c6c2-5791-478a-bfa2-c98fedc041ed-kube-api-access-rprrp\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.114873 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.114809    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7xzf\" (UniqueName: \"kubernetes.io/projected/33a822a2-33ea-4a2c-93e8-383364e7593d-kube-api-access-t7xzf\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.114873 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.114839    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb57c6c2-5791-478a-bfa2-c98fedc041ed-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.114873 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.114865    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.115070 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.114920    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.115070 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.114953    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.115070 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.114997    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.115070 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.115015    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.115070 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.115045    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.115354 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.115083    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.115354 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.115155    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.116123 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.116101    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.116768 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.116709    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.117550 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.117522    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.117707 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.117678    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb57c6c2-5791-478a-bfa2-c98fedc041ed-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.118419 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.118391    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.118520 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.118503    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.118780 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.118756    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.119248 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.119222    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.119386 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.119370    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.120623 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.120597    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.120720 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.120675    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb57c6c2-5791-478a-bfa2-c98fedc041ed-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.137545 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.137525    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rprrp\" (UniqueName: \"kubernetes.io/projected/fb57c6c2-5791-478a-bfa2-c98fedc041ed-kube-api-access-rprrp\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:43.217386 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.217362    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.217490 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.217398    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33a822a2-33ea-4a2c-93e8-383364e7593d-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.217490 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.217415    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33a822a2-33ea-4a2c-93e8-383364e7593d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.217613 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.217528    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.217613 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.217560    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/33a822a2-33ea-4a2c-93e8-383364e7593d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.217613 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.217594    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7xzf\" (UniqueName: \"kubernetes.io/projected/33a822a2-33ea-4a2c-93e8-383364e7593d-kube-api-access-t7xzf\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.217785 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.217620    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.217964 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.217933    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/33a822a2-33ea-4a2c-93e8-383364e7593d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.219765 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.219741    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.220075 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.220048    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.220151 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.220092    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33a822a2-33ea-4a2c-93e8-383364e7593d-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.220219 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.220161    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.220261 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.220215    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33a822a2-33ea-4a2c-93e8-383364e7593d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.238035 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.238014    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7xzf\" (UniqueName: \"kubernetes.io/projected/33a822a2-33ea-4a2c-93e8-383364e7593d-kube-api-access-t7xzf\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.502190 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.502102    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1524f4c0-dd9a-4c2b-811e-4e10af5a62fd" path="/var/lib/kubelet/pods/1524f4c0-dd9a-4c2b-811e-4e10af5a62fd/volumes"
Oct 30 03:47:43.502538 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.502524    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d53f2afb-6b3d-4290-86f3-c663ef9734be" path="/var/lib/kubelet/pods/d53f2afb-6b3d-4290-86f3-c663ef9734be/volumes"
Oct 30 03:47:43.533445 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.533416    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:43.851500 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:43.851468    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 03:47:43.852218 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:47:43.852171    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a822a2_33ea_4a2c_93e8_383364e7593d.slice/crio-327defd8dab7543809c3563cc8fef9642b803f889046af84a540ff2c82c1e520 WatchSource:0}: Error finding container 327defd8dab7543809c3563cc8fef9642b803f889046af84a540ff2c82c1e520: Status 404 returned error can't find the container with id 327defd8dab7543809c3563cc8fef9642b803f889046af84a540ff2c82c1e520
Oct 30 03:47:44.250098 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:44.250063    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"33a822a2-33ea-4a2c-93e8-383364e7593d","Type":"ContainerStarted","Data":"87db361423cd2c3402715282c32b09f56510f6d5b66aaea48da1b1a55bea93bc"}
Oct 30 03:47:44.250487 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:44.250121    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"33a822a2-33ea-4a2c-93e8-383364e7593d","Type":"ContainerStarted","Data":"327defd8dab7543809c3563cc8fef9642b803f889046af84a540ff2c82c1e520"}
Oct 30 03:47:48.269923 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:48.269889    4960 generic.go:356] "Generic (PLEG): container finished" podID="33a822a2-33ea-4a2c-93e8-383364e7593d" containerID="87db361423cd2c3402715282c32b09f56510f6d5b66aaea48da1b1a55bea93bc" exitCode=0
Oct 30 03:47:48.270301 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:48.269956    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"33a822a2-33ea-4a2c-93e8-383364e7593d","Type":"ContainerDied","Data":"87db361423cd2c3402715282c32b09f56510f6d5b66aaea48da1b1a55bea93bc"}
Oct 30 03:47:48.562750 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:48.562674    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:48.593984 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:48.593951    4960 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:48.663382 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:48.663357    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:48.663466 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:48.663458    4960 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:48.679104 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:48.679079    4960 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") DevicePath \"csi-6f8c8e4c1326657251c30594036033694cadf25e97408e4f9d9da436f78f3fa4\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:48.783680 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:48.783652    4960 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/8c516923ea0339579ebf808cef81b9e0aaf55e5b619e2c6187025fa3b2b51f7f/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:48.811815 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:48.811790    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:48.893284 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:48.893256    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:47:49.214590 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:49.214566    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:47:49.215555 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:47:49.215527    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb57c6c2_5791_478a_bfa2_c98fedc041ed.slice/crio-e0f3a425a761506b268b83ec6b7e33de61b8b4d5c775c2df0d5a07e6d595e1f2 WatchSource:0}: Error finding container e0f3a425a761506b268b83ec6b7e33de61b8b4d5c775c2df0d5a07e6d595e1f2: Status 404 returned error can't find the container with id e0f3a425a761506b268b83ec6b7e33de61b8b4d5c775c2df0d5a07e6d595e1f2
Oct 30 03:47:49.277136 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:49.277095    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"33a822a2-33ea-4a2c-93e8-383364e7593d","Type":"ContainerStarted","Data":"1ae073d5f0e640139548e908ef7b4c712486362710b8a896508596acb7116215"}
Oct 30 03:47:49.277483 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:49.277171    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"33a822a2-33ea-4a2c-93e8-383364e7593d","Type":"ContainerStarted","Data":"36ee9f2f0cf8f33f35092439422ee951d08c265842ffc2569646213a3a48a152"}
Oct 30 03:47:49.277483 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:49.277406    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:49.278436 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:49.278412    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fb57c6c2-5791-478a-bfa2-c98fedc041ed","Type":"ContainerStarted","Data":"e0f3a425a761506b268b83ec6b7e33de61b8b4d5c775c2df0d5a07e6d595e1f2"}
Oct 30 03:47:49.327466 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:49.327424    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.941914842 podStartE2EDuration="7.327414227s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.385499385s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:49.325892679 +0000 UTC m=+3949.974941100" watchObservedRunningTime="2026-10-30 03:47:49.327414227 +0000 UTC m=+3949.976462644"
Oct 30 03:47:50.221564 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:50.221535    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"
Oct 30 03:47:50.285791 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:50.285754    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fb57c6c2-5791-478a-bfa2-c98fedc041ed","Type":"ContainerStarted","Data":"baa26f5b8c7a1b5429b257fa15a9166778845dee683a19d418b172c672ca33fc"}
Oct 30 03:47:53.301618 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:53.301541    4960 generic.go:356] "Generic (PLEG): container finished" podID="fb57c6c2-5791-478a-bfa2-c98fedc041ed" containerID="baa26f5b8c7a1b5429b257fa15a9166778845dee683a19d418b172c672ca33fc" exitCode=0
Oct 30 03:47:53.301952 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:53.301614    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fb57c6c2-5791-478a-bfa2-c98fedc041ed","Type":"ContainerDied","Data":"baa26f5b8c7a1b5429b257fa15a9166778845dee683a19d418b172c672ca33fc"}
Oct 30 03:47:53.952924 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:47:53.952884    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:47:54.309553 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:54.309518    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fb57c6c2-5791-478a-bfa2-c98fedc041ed","Type":"ContainerStarted","Data":"cf2b5d51ee3a9449fd8f7f34bf24de2fcc39c09d6bd5468fc8590d863c5d4673"}
Oct 30 03:47:54.309912 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:54.309612    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fb57c6c2-5791-478a-bfa2-c98fedc041ed","Type":"ContainerStarted","Data":"4d3c374d00c1fea545c169e50f690bd7195dc60ef189082375d4841c328fe6d4"}
Oct 30 03:47:54.309912 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:54.309638    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fb57c6c2-5791-478a-bfa2-c98fedc041ed","Type":"ContainerStarted","Data":"c53458936c2a39d594173225ff5151c76a44e407fa9e425c80dcef158bb6b9c2"}
Oct 30 03:47:54.391394 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:54.391345    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=9.106940138 podStartE2EDuration="12.391333281s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.284393143s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:54.376878551 +0000 UTC m=+3955.025926985" watchObservedRunningTime="2026-10-30 03:47:54.391333281 +0000 UTC m=+3955.040381727"
Oct 30 03:47:55.287157 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:55.287126    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:47:57.078141 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:57.078109    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:58.894048 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:47:58.894017    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:48:03.894214 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:03.894160    4960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:48:03.899342 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:03.899320    4960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:48:04.361058 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:04.361033    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:48:12.708605 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:12.708552    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:12.814313 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:12.814283    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:12.919711 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:12.919675    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:13.033670 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:13.033605    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:14.491997 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:14.491966    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:48:14.492430 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:14.492195    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="a01e9f19-e49e-4387-a8d2-d651a6120590" containerName="otc-container" containerID="cri-o://ce16dfbe7d4807b5dbbfa284861b669b8b4dc94505ed329eed2e4a8b4c99f4dc" gracePeriod=30
Oct 30 03:48:14.626880 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:14.626851    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:14.743151 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:14.743081    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:14.855122 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:14.855089    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:16.847751 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:16.847726    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:16.983970 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:16.983899    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-data-science-collector-ta-client-cert\") pod \"a01e9f19-e49e-4387-a8d2-d651a6120590\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") "
Oct 30 03:48:16.983970 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:16.983952    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a01e9f19-e49e-4387-a8d2-d651a6120590-otc-internal\" (UniqueName: \"kubernetes.io/configmap/a01e9f19-e49e-4387-a8d2-d651a6120590-otc-internal\") pod \"a01e9f19-e49e-4387-a8d2-d651a6120590\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") "
Oct 30 03:48:16.984142 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:16.984074    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a01e9f19-e49e-4387-a8d2-d651a6120590-kube-api-access-fwbtb\" (UniqueName: \"kubernetes.io/projected/a01e9f19-e49e-4387-a8d2-d651a6120590-kube-api-access-fwbtb\") pod \"a01e9f19-e49e-4387-a8d2-d651a6120590\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") "
Oct 30 03:48:16.984142 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:16.984133    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-tls-certs\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-tls-certs\") pod \"a01e9f19-e49e-4387-a8d2-d651a6120590\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") "
Oct 30 03:48:16.984273 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:16.984257    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-monitoring-tls-certs\") pod \"a01e9f19-e49e-4387-a8d2-d651a6120590\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") "
Oct 30 03:48:16.984327 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:16.984288    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-otlp-tls-certs\") pod \"a01e9f19-e49e-4387-a8d2-d651a6120590\" (UID: \"a01e9f19-e49e-4387-a8d2-d651a6120590\") "
Oct 30 03:48:16.984381 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:16.984354    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a01e9f19-e49e-4387-a8d2-d651a6120590-otc-internal" pod "a01e9f19-e49e-4387-a8d2-d651a6120590" (UID: "a01e9f19-e49e-4387-a8d2-d651a6120590"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:16.984575 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:16.984554    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a01e9f19-e49e-4387-a8d2-d651a6120590-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:16.986478 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:16.986449    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-data-science-collector-ta-client-cert" pod "a01e9f19-e49e-4387-a8d2-d651a6120590" (UID: "a01e9f19-e49e-4387-a8d2-d651a6120590"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:16.986478 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:16.986462    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-monitoring-tls-certs" pod "a01e9f19-e49e-4387-a8d2-d651a6120590" (UID: "a01e9f19-e49e-4387-a8d2-d651a6120590"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:16.986670 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:16.986657    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-otlp-tls-certs" pod "a01e9f19-e49e-4387-a8d2-d651a6120590" (UID: "a01e9f19-e49e-4387-a8d2-d651a6120590"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:16.986921 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:16.986899    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01e9f19-e49e-4387-a8d2-d651a6120590-kube-api-access-fwbtb" pod "a01e9f19-e49e-4387-a8d2-d651a6120590" (UID: "a01e9f19-e49e-4387-a8d2-d651a6120590"). InnerVolumeSpecName "kube-api-access-fwbtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:48:16.986921 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:16.986905    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-tls-certs" pod "a01e9f19-e49e-4387-a8d2-d651a6120590" (UID: "a01e9f19-e49e-4387-a8d2-d651a6120590"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:17.085376 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.085342    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwbtb\" (UniqueName: \"kubernetes.io/projected/a01e9f19-e49e-4387-a8d2-d651a6120590-kube-api-access-fwbtb\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:17.085376 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.085374    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:17.085376 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.085384    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:17.085567 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.085393    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:17.085567 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.085404    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a01e9f19-e49e-4387-a8d2-d651a6120590-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:17.421794 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.421763    4960 generic.go:356] "Generic (PLEG): container finished" podID="a01e9f19-e49e-4387-a8d2-d651a6120590" containerID="ce16dfbe7d4807b5dbbfa284861b669b8b4dc94505ed329eed2e4a8b4c99f4dc" exitCode=0
Oct 30 03:48:17.421974 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.421810    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.421974 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.421853    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a01e9f19-e49e-4387-a8d2-d651a6120590","Type":"ContainerDied","Data":"ce16dfbe7d4807b5dbbfa284861b669b8b4dc94505ed329eed2e4a8b4c99f4dc"}
Oct 30 03:48:17.421974 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.421939    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a01e9f19-e49e-4387-a8d2-d651a6120590","Type":"ContainerDied","Data":"9acd7a085c85791e9346a8830eb92ce8d286a5a87f3df10f93f62b4bb0239194"}
Oct 30 03:48:17.421974 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.421972    4960 scope.go:122] "RemoveContainer" containerID="ce16dfbe7d4807b5dbbfa284861b669b8b4dc94505ed329eed2e4a8b4c99f4dc"
Oct 30 03:48:17.431639 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.431619    4960 scope.go:122] "RemoveContainer" containerID="ce16dfbe7d4807b5dbbfa284861b669b8b4dc94505ed329eed2e4a8b4c99f4dc"
Oct 30 03:48:17.431884 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:48:17.431865    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce16dfbe7d4807b5dbbfa284861b669b8b4dc94505ed329eed2e4a8b4c99f4dc\": container with ID starting with ce16dfbe7d4807b5dbbfa284861b669b8b4dc94505ed329eed2e4a8b4c99f4dc not found: ID does not exist" containerID="ce16dfbe7d4807b5dbbfa284861b669b8b4dc94505ed329eed2e4a8b4c99f4dc"
Oct 30 03:48:17.431927 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.431896    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce16dfbe7d4807b5dbbfa284861b669b8b4dc94505ed329eed2e4a8b4c99f4dc"} err="failed to get container status \"ce16dfbe7d4807b5dbbfa284861b669b8b4dc94505ed329eed2e4a8b4c99f4dc\": rpc error: code = NotFound desc = could not find container \"ce16dfbe7d4807b5dbbfa284861b669b8b4dc94505ed329eed2e4a8b4c99f4dc\": container with ID starting with ce16dfbe7d4807b5dbbfa284861b669b8b4dc94505ed329eed2e4a8b4c99f4dc not found: ID does not exist"
Oct 30 03:48:17.475504 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.475479    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:48:17.489313 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.489291    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:48:17.503131 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.503098    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a01e9f19-e49e-4387-a8d2-d651a6120590" path="/var/lib/kubelet/pods/a01e9f19-e49e-4387-a8d2-d651a6120590/volumes"
Oct 30 03:48:17.515097 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.515077    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:48:17.515459 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.515446    4960 cpu_manager.go:414] "containerMap: removing container" podUID="a01e9f19-e49e-4387-a8d2-d651a6120590" containerName="otc-container"
Oct 30 03:48:17.515511 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.515461    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a01e9f19-e49e-4387-a8d2-d651a6120590" containerName="otc-container"
Oct 30 03:48:17.515561 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.515551    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="a01e9f19-e49e-4387-a8d2-d651a6120590" containerName="otc-container"
Oct 30 03:48:17.519904 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.519888    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.538132 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.537853    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:48:17.538132 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.537956    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:48:17.538132 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.537961    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wtzv9\""
Oct 30 03:48:17.538132 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.538017    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:48:17.538132 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.538060    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:48:17.538448 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.538287    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:48:17.539091 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.539073    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:48:17.691154 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.691071    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.691320 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.691208    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjflv\" (UniqueName: \"kubernetes.io/projected/74a07755-82ec-4ba1-895d-f2676e0bc795-kube-api-access-tjflv\") pod \"data-science-collector-collector-1\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.691320 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.691248    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.691320 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.691312    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/74a07755-82ec-4ba1-895d-f2676e0bc795-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.691478 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.691337    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.691478 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.691450    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.792480 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.792455    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjflv\" (UniqueName: \"kubernetes.io/projected/74a07755-82ec-4ba1-895d-f2676e0bc795-kube-api-access-tjflv\") pod \"data-science-collector-collector-1\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.792612 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.792490    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.792612 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.792506    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/74a07755-82ec-4ba1-895d-f2676e0bc795-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.792612 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.792520    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.792612 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.792570    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.792612 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.792594    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.793522 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.793405    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/74a07755-82ec-4ba1-895d-f2676e0bc795-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.795198 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.795161    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.795355 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.795334    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.795612 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.795592    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.795679 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.795611    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.813036 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.813013    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjflv\" (UniqueName: \"kubernetes.io/projected/74a07755-82ec-4ba1-895d-f2676e0bc795-kube-api-access-tjflv\") pod \"data-science-collector-collector-1\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:17.831971 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:17.831947    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:18.125288 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:18.125252    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:48:18.126410 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:48:18.126385    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a07755_82ec_4ba1_895d_f2676e0bc795.slice/crio-bb983e59c95b6122cb21f34f613f103332a20c1001746ef2c89c0ae2d2451ba1 WatchSource:0}: Error finding container bb983e59c95b6122cb21f34f613f103332a20c1001746ef2c89c0ae2d2451ba1: Status 404 returned error can't find the container with id bb983e59c95b6122cb21f34f613f103332a20c1001746ef2c89c0ae2d2451ba1
Oct 30 03:48:18.128642 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:18.128613    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:48:18.128715 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:18.128680    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:48:18.428469 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:18.428382    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"74a07755-82ec-4ba1-895d-f2676e0bc795","Type":"ContainerStarted","Data":"8bb5feded6a2e80feaa01d634d36380326dc92ba25324765f229ca8c2c90fe7f"}
Oct 30 03:48:18.428469 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:18.428445    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"74a07755-82ec-4ba1-895d-f2676e0bc795","Type":"ContainerStarted","Data":"bb983e59c95b6122cb21f34f613f103332a20c1001746ef2c89c0ae2d2451ba1"}
Oct 30 03:48:18.477962 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:18.477910    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.477896768 podStartE2EDuration="1.477896768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:48:18.476958892 +0000 UTC m=+3979.126007313" watchObservedRunningTime="2026-10-30 03:48:18.477896768 +0000 UTC m=+3979.126945189"
Oct 30 03:48:23.953231 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:48:23.953165    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:48:25.642371 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:25.642338    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:25.774383 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:25.774341    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:33.810310 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:33.810275    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:37.821344 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:37.821305    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:37.934083 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:37.934041    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:38.052635 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:38.052572    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:38.132228 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:38.132135    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"]
Oct 30 03:48:38.135577 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:38.135559    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"
Oct 30 03:48:38.147808 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:38.147789    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0ebf1223-c032-4175-bfad-07d478f9e689-tls\") pod \"korrel8r-66d6f7f696-cs62r\" (UID: \"0ebf1223-c032-4175-bfad-07d478f9e689\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"
Oct 30 03:48:38.147910 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:38.147838    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ebf1223-c032-4175-bfad-07d478f9e689-config\") pod \"korrel8r-66d6f7f696-cs62r\" (UID: \"0ebf1223-c032-4175-bfad-07d478f9e689\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"
Oct 30 03:48:38.147910 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:38.147856    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpd2n\" (UniqueName: \"kubernetes.io/projected/0ebf1223-c032-4175-bfad-07d478f9e689-kube-api-access-cpd2n\") pod \"korrel8r-66d6f7f696-cs62r\" (UID: \"0ebf1223-c032-4175-bfad-07d478f9e689\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"
Oct 30 03:48:38.154784 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:38.154759    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"]
Oct 30 03:48:38.248614 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:38.248589    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ebf1223-c032-4175-bfad-07d478f9e689-config\") pod \"korrel8r-66d6f7f696-cs62r\" (UID: \"0ebf1223-c032-4175-bfad-07d478f9e689\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"
Oct 30 03:48:38.248790 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:38.248620    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpd2n\" (UniqueName: \"kubernetes.io/projected/0ebf1223-c032-4175-bfad-07d478f9e689-kube-api-access-cpd2n\") pod \"korrel8r-66d6f7f696-cs62r\" (UID: \"0ebf1223-c032-4175-bfad-07d478f9e689\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"
Oct 30 03:48:38.248790 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:38.248672    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0ebf1223-c032-4175-bfad-07d478f9e689-tls\") pod \"korrel8r-66d6f7f696-cs62r\" (UID: \"0ebf1223-c032-4175-bfad-07d478f9e689\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"
Oct 30 03:48:38.249446 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:38.249425    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ebf1223-c032-4175-bfad-07d478f9e689-config\") pod \"korrel8r-66d6f7f696-cs62r\" (UID: \"0ebf1223-c032-4175-bfad-07d478f9e689\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"
Oct 30 03:48:38.251418 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:38.251389    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0ebf1223-c032-4175-bfad-07d478f9e689-tls\") pod \"korrel8r-66d6f7f696-cs62r\" (UID: \"0ebf1223-c032-4175-bfad-07d478f9e689\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"
Oct 30 03:48:38.268548 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:38.268526    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpd2n\" (UniqueName: \"kubernetes.io/projected/0ebf1223-c032-4175-bfad-07d478f9e689-kube-api-access-cpd2n\") pod \"korrel8r-66d6f7f696-cs62r\" (UID: \"0ebf1223-c032-4175-bfad-07d478f9e689\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"
Oct 30 03:48:38.449009 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:38.448944    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"
Oct 30 03:48:38.740463 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:38.740435    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"]
Oct 30 03:48:38.742252 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:48:38.742227    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ebf1223_c032_4175_bfad_07d478f9e689.slice/crio-1c1f5a96295d9837804409d66d583b7247b000be4a50c14472eedc5cb8a91dfb WatchSource:0}: Error finding container 1c1f5a96295d9837804409d66d583b7247b000be4a50c14472eedc5cb8a91dfb: Status 404 returned error can't find the container with id 1c1f5a96295d9837804409d66d583b7247b000be4a50c14472eedc5cb8a91dfb
Oct 30 03:48:39.523711 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:39.523677    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r" event={"ID":"0ebf1223-c032-4175-bfad-07d478f9e689","Type":"ContainerStarted","Data":"f97881ba9c043d24de61d776b990912e75365c932b7a107ef455cdad544d2bbb"}
Oct 30 03:48:39.524114 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:39.523736    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"
Oct 30 03:48:39.524114 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:39.523760    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r" event={"ID":"0ebf1223-c032-4175-bfad-07d478f9e689","Type":"ContainerStarted","Data":"1c1f5a96295d9837804409d66d583b7247b000be4a50c14472eedc5cb8a91dfb"}
Oct 30 03:48:39.566245 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:39.566173    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r" podStartSLOduration=1.5661572000000001 podStartE2EDuration="1.5661572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:48:39.565604619 +0000 UTC m=+4000.214653040" watchObservedRunningTime="2026-10-30 03:48:39.5661572 +0000 UTC m=+4000.215205621"
Oct 30 03:48:39.653230 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:39.653172    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:48:39.653397 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:39.653377    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="74a07755-82ec-4ba1-895d-f2676e0bc795" containerName="otc-container" containerID="cri-o://8bb5feded6a2e80feaa01d634d36380326dc92ba25324765f229ca8c2c90fe7f" gracePeriod=30
Oct 30 03:48:39.666229 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:39.666155    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:39.798416 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:39.798344    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:39.931727 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:39.931691    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:40.905426 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:40.905402    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:40.971729 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:40.971700    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-monitoring-tls-certs\") pod \"74a07755-82ec-4ba1-895d-f2676e0bc795\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") "
Oct 30 03:48:40.971878 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:40.971750    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-tls-certs\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-tls-certs\") pod \"74a07755-82ec-4ba1-895d-f2676e0bc795\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") "
Oct 30 03:48:40.971940 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:40.971870    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a07755-82ec-4ba1-895d-f2676e0bc795-otc-internal\" (UniqueName: \"kubernetes.io/configmap/74a07755-82ec-4ba1-895d-f2676e0bc795-otc-internal\") pod \"74a07755-82ec-4ba1-895d-f2676e0bc795\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") "
Oct 30 03:48:40.972005 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:40.971987    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-otlp-tls-certs\") pod \"74a07755-82ec-4ba1-895d-f2676e0bc795\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") "
Oct 30 03:48:40.972073 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:40.972046    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-data-science-collector-ta-client-cert\") pod \"74a07755-82ec-4ba1-895d-f2676e0bc795\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") "
Oct 30 03:48:40.972137 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:40.972089    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a07755-82ec-4ba1-895d-f2676e0bc795-kube-api-access-tjflv\" (UniqueName: \"kubernetes.io/projected/74a07755-82ec-4ba1-895d-f2676e0bc795-kube-api-access-tjflv\") pod \"74a07755-82ec-4ba1-895d-f2676e0bc795\" (UID: \"74a07755-82ec-4ba1-895d-f2676e0bc795\") "
Oct 30 03:48:40.972358 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:40.972328    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a07755-82ec-4ba1-895d-f2676e0bc795-otc-internal" pod "74a07755-82ec-4ba1-895d-f2676e0bc795" (UID: "74a07755-82ec-4ba1-895d-f2676e0bc795"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:40.974606 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:40.974567    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-tls-certs" pod "74a07755-82ec-4ba1-895d-f2676e0bc795" (UID: "74a07755-82ec-4ba1-895d-f2676e0bc795"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:40.974606 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:40.974598    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a07755-82ec-4ba1-895d-f2676e0bc795-kube-api-access-tjflv" pod "74a07755-82ec-4ba1-895d-f2676e0bc795" (UID: "74a07755-82ec-4ba1-895d-f2676e0bc795"). InnerVolumeSpecName "kube-api-access-tjflv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:48:40.974837 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:40.974816    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-monitoring-tls-certs" pod "74a07755-82ec-4ba1-895d-f2676e0bc795" (UID: "74a07755-82ec-4ba1-895d-f2676e0bc795"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:40.974960 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:40.974944    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-otlp-tls-certs" pod "74a07755-82ec-4ba1-895d-f2676e0bc795" (UID: "74a07755-82ec-4ba1-895d-f2676e0bc795"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:40.975213 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:40.975169    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-data-science-collector-ta-client-cert" pod "74a07755-82ec-4ba1-895d-f2676e0bc795" (UID: "74a07755-82ec-4ba1-895d-f2676e0bc795"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:41.080582 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.076609    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjflv\" (UniqueName: \"kubernetes.io/projected/74a07755-82ec-4ba1-895d-f2676e0bc795-kube-api-access-tjflv\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:41.080582 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.076648    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:41.080582 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.076665    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:41.080582 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.076689    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/74a07755-82ec-4ba1-895d-f2676e0bc795-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:41.080582 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.076703    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:41.080582 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.076718    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/74a07755-82ec-4ba1-895d-f2676e0bc795-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:41.536218 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.536166    4960 generic.go:356] "Generic (PLEG): container finished" podID="74a07755-82ec-4ba1-895d-f2676e0bc795" containerID="8bb5feded6a2e80feaa01d634d36380326dc92ba25324765f229ca8c2c90fe7f" exitCode=0
Oct 30 03:48:41.536372 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.536221    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"74a07755-82ec-4ba1-895d-f2676e0bc795","Type":"ContainerDied","Data":"8bb5feded6a2e80feaa01d634d36380326dc92ba25324765f229ca8c2c90fe7f"}
Oct 30 03:48:41.536372 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.536256    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.536372 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.536274    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"74a07755-82ec-4ba1-895d-f2676e0bc795","Type":"ContainerDied","Data":"bb983e59c95b6122cb21f34f613f103332a20c1001746ef2c89c0ae2d2451ba1"}
Oct 30 03:48:41.536372 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.536295    4960 scope.go:122] "RemoveContainer" containerID="8bb5feded6a2e80feaa01d634d36380326dc92ba25324765f229ca8c2c90fe7f"
Oct 30 03:48:41.544773 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.544758    4960 scope.go:122] "RemoveContainer" containerID="8bb5feded6a2e80feaa01d634d36380326dc92ba25324765f229ca8c2c90fe7f"
Oct 30 03:48:41.545060 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:48:41.545033    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bb5feded6a2e80feaa01d634d36380326dc92ba25324765f229ca8c2c90fe7f\": container with ID starting with 8bb5feded6a2e80feaa01d634d36380326dc92ba25324765f229ca8c2c90fe7f not found: ID does not exist" containerID="8bb5feded6a2e80feaa01d634d36380326dc92ba25324765f229ca8c2c90fe7f"
Oct 30 03:48:41.545120 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.545068    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb5feded6a2e80feaa01d634d36380326dc92ba25324765f229ca8c2c90fe7f"} err="failed to get container status \"8bb5feded6a2e80feaa01d634d36380326dc92ba25324765f229ca8c2c90fe7f\": rpc error: code = NotFound desc = could not find container \"8bb5feded6a2e80feaa01d634d36380326dc92ba25324765f229ca8c2c90fe7f\": container with ID starting with 8bb5feded6a2e80feaa01d634d36380326dc92ba25324765f229ca8c2c90fe7f not found: ID does not exist"
Oct 30 03:48:41.600423 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.600396    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:48:41.613760 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.613737    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:48:41.697253 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.697230    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:48:41.697580 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.697568    4960 cpu_manager.go:414] "containerMap: removing container" podUID="74a07755-82ec-4ba1-895d-f2676e0bc795" containerName="otc-container"
Oct 30 03:48:41.697627 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.697582    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a07755-82ec-4ba1-895d-f2676e0bc795" containerName="otc-container"
Oct 30 03:48:41.697664 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.697646    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a07755-82ec-4ba1-895d-f2676e0bc795" containerName="otc-container"
Oct 30 03:48:41.701871 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.701856    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.724605 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.724575    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:48:41.724725 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.724606    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wtzv9\""
Oct 30 03:48:41.724725 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.724643    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:48:41.732584 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.732564    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:48:41.733441 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.733426    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\""
Oct 30 03:48:41.737820 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.737801    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:48:41.756344 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.756322    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:48:41.782891 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.782840    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.782996 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.782890    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.782996 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.782962    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4fcc010f-9f67-4997-b009-c1330d4a01f4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.783082 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.783032    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.783131 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.783082    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.783131 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.783107    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4zx6\" (UniqueName: \"kubernetes.io/projected/4fcc010f-9f67-4997-b009-c1330d4a01f4-kube-api-access-j4zx6\") pod \"data-science-collector-collector-1\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.884441 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.884418    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.884532 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.884446    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.884532 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.884470    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4zx6\" (UniqueName: \"kubernetes.io/projected/4fcc010f-9f67-4997-b009-c1330d4a01f4-kube-api-access-j4zx6\") pod \"data-science-collector-collector-1\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.884626 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.884608    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.884686 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.884665    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.884747 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.884698    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4fcc010f-9f67-4997-b009-c1330d4a01f4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.885401 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.885380    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4fcc010f-9f67-4997-b009-c1330d4a01f4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.887281 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.887253    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.887281 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.887267    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.887777 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.887758    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.887813 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.887775    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.913302 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.913273    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4zx6\" (UniqueName: \"kubernetes.io/projected/4fcc010f-9f67-4997-b009-c1330d4a01f4-kube-api-access-j4zx6\") pod \"data-science-collector-collector-1\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:41.930483 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.930460    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:48:41.934265 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.934250    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:41.965469 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.965446    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 03:48:41.965601 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.965489    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 03:48:41.965849 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.965824    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 03:48:41.965849 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.965830    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:48:41.970174 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.970157    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 03:48:41.970174 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.970159    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:48:41.970349 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.970333    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-kj672\""
Oct 30 03:48:41.970349 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.970339    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 03:48:41.979535 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:41.979512    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:48:42.014579 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.014560    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:42.033709 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.033657    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:42.087168 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.087134    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.087338 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.087225    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk8sm\" (UniqueName: \"kubernetes.io/projected/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-kube-api-access-kk8sm\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.087338 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.087310    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.087456 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.087341    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.087456 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.087369    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.087456 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.087415    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.087609 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.087452    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.087609 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.087482    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.184146 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.184121    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:42.188274 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.188254    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.188350 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.188282    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.188350 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.188307    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.188350 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.188335    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.188495 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.188360    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.188495 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.188380    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.188495 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.188414    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.188495 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.188471    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk8sm\" (UniqueName: \"kubernetes.io/projected/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-kube-api-access-kk8sm\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.189168 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.189146    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.189291 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.189172    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.189343 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.189281    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.189343 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.189321    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.190986 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.190959    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.190986 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.190978    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.191102 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.191041    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.212334 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.212314    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk8sm\" (UniqueName: \"kubernetes.io/projected/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-kube-api-access-kk8sm\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:42.312366 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.312309    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:48:42.315166 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:48:42.315142    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fcc010f_9f67_4997_b009_c1330d4a01f4.slice/crio-9a3a4d94a21d85400d0321084f6369360d2fcc20a5a6c3e07c778d8b278bde00 WatchSource:0}: Error finding container 9a3a4d94a21d85400d0321084f6369360d2fcc20a5a6c3e07c778d8b278bde00: Status 404 returned error can't find the container with id 9a3a4d94a21d85400d0321084f6369360d2fcc20a5a6c3e07c778d8b278bde00
Oct 30 03:48:42.317470 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.317438    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:48:42.317570 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.317497    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:48:42.544613 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.544581    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4fcc010f-9f67-4997-b009-c1330d4a01f4","Type":"ContainerStarted","Data":"ce6b86243c19b62761b644afe4e33c04d44f021af3c2c3bc0899339eb2ab2e00"}
Oct 30 03:48:42.544758 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.544629    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4fcc010f-9f67-4997-b009-c1330d4a01f4","Type":"ContainerStarted","Data":"9a3a4d94a21d85400d0321084f6369360d2fcc20a5a6c3e07c778d8b278bde00"}
Oct 30 03:48:42.591317 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:42.591265    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.591248034 podStartE2EDuration="1.591248034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:48:42.590172979 +0000 UTC m=+4003.239221414" watchObservedRunningTime="2026-10-30 03:48:42.591248034 +0000 UTC m=+4003.240296456"
Oct 30 03:48:43.502796 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:43.502764    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a07755-82ec-4ba1-895d-f2676e0bc795" path="/var/lib/kubelet/pods/74a07755-82ec-4ba1-895d-f2676e0bc795/volumes"
Oct 30 03:48:47.323258 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:47.323220    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:47.520618 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:47.520585    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:47.632435 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:47.632374    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:47.707547 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:47.707519    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"]
Oct 30 03:48:47.707740 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:47.707718    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r" podUID="0ebf1223-c032-4175-bfad-07d478f9e689" containerName="korrel8r" containerID="cri-o://f97881ba9c043d24de61d776b990912e75365c932b7a107ef455cdad544d2bbb" gracePeriod=30
Oct 30 03:48:47.709284 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:47.709263    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"
Oct 30 03:48:47.945562 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:47.945540    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"
Oct 30 03:48:48.034773 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:48.034734    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebf1223-c032-4175-bfad-07d478f9e689-kube-api-access-cpd2n\" (UniqueName: \"kubernetes.io/projected/0ebf1223-c032-4175-bfad-07d478f9e689-kube-api-access-cpd2n\") pod \"0ebf1223-c032-4175-bfad-07d478f9e689\" (UID: \"0ebf1223-c032-4175-bfad-07d478f9e689\") "
Oct 30 03:48:48.034912 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:48.034827    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ebf1223-c032-4175-bfad-07d478f9e689-config\" (UniqueName: \"kubernetes.io/configmap/0ebf1223-c032-4175-bfad-07d478f9e689-config\") pod \"0ebf1223-c032-4175-bfad-07d478f9e689\" (UID: \"0ebf1223-c032-4175-bfad-07d478f9e689\") "
Oct 30 03:48:48.034912 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:48.034849    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ebf1223-c032-4175-bfad-07d478f9e689-tls\" (UniqueName: \"kubernetes.io/secret/0ebf1223-c032-4175-bfad-07d478f9e689-tls\") pod \"0ebf1223-c032-4175-bfad-07d478f9e689\" (UID: \"0ebf1223-c032-4175-bfad-07d478f9e689\") "
Oct 30 03:48:48.035249 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:48.035215    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ebf1223-c032-4175-bfad-07d478f9e689-config" pod "0ebf1223-c032-4175-bfad-07d478f9e689" (UID: "0ebf1223-c032-4175-bfad-07d478f9e689"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:48.037111 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:48.037090    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebf1223-c032-4175-bfad-07d478f9e689-kube-api-access-cpd2n" pod "0ebf1223-c032-4175-bfad-07d478f9e689" (UID: "0ebf1223-c032-4175-bfad-07d478f9e689"). InnerVolumeSpecName "kube-api-access-cpd2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:48:48.037197 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:48.037130    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ebf1223-c032-4175-bfad-07d478f9e689-tls" pod "0ebf1223-c032-4175-bfad-07d478f9e689" (UID: "0ebf1223-c032-4175-bfad-07d478f9e689"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:48.136104 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:48.136080    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ebf1223-c032-4175-bfad-07d478f9e689-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:48.136104 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:48.136100    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0ebf1223-c032-4175-bfad-07d478f9e689-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:48.136271 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:48.136110    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpd2n\" (UniqueName: \"kubernetes.io/projected/0ebf1223-c032-4175-bfad-07d478f9e689-kube-api-access-cpd2n\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:48.572727 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:48.572695    4960 generic.go:356] "Generic (PLEG): container finished" podID="0ebf1223-c032-4175-bfad-07d478f9e689" containerID="f97881ba9c043d24de61d776b990912e75365c932b7a107ef455cdad544d2bbb" exitCode=0
Oct 30 03:48:48.573136 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:48.572744    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"
Oct 30 03:48:48.573136 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:48.572789    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r" event={"ID":"0ebf1223-c032-4175-bfad-07d478f9e689","Type":"ContainerDied","Data":"f97881ba9c043d24de61d776b990912e75365c932b7a107ef455cdad544d2bbb"}
Oct 30 03:48:48.573136 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:48.572859    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r" event={"ID":"0ebf1223-c032-4175-bfad-07d478f9e689","Type":"ContainerDied","Data":"1c1f5a96295d9837804409d66d583b7247b000be4a50c14472eedc5cb8a91dfb"}
Oct 30 03:48:48.573136 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:48.572893    4960 scope.go:122] "RemoveContainer" containerID="f97881ba9c043d24de61d776b990912e75365c932b7a107ef455cdad544d2bbb"
Oct 30 03:48:48.581912 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:48.581896    4960 scope.go:122] "RemoveContainer" containerID="f97881ba9c043d24de61d776b990912e75365c932b7a107ef455cdad544d2bbb"
Oct 30 03:48:48.582210 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:48:48.582165    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f97881ba9c043d24de61d776b990912e75365c932b7a107ef455cdad544d2bbb\": container with ID starting with f97881ba9c043d24de61d776b990912e75365c932b7a107ef455cdad544d2bbb not found: ID does not exist" containerID="f97881ba9c043d24de61d776b990912e75365c932b7a107ef455cdad544d2bbb"
Oct 30 03:48:48.582272 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:48.582223    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f97881ba9c043d24de61d776b990912e75365c932b7a107ef455cdad544d2bbb"} err="failed to get container status \"f97881ba9c043d24de61d776b990912e75365c932b7a107ef455cdad544d2bbb\": rpc error: code = NotFound desc = could not find container \"f97881ba9c043d24de61d776b990912e75365c932b7a107ef455cdad544d2bbb\": container with ID starting with f97881ba9c043d24de61d776b990912e75365c932b7a107ef455cdad544d2bbb not found: ID does not exist"
Oct 30 03:48:48.627159 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:48.627134    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"]
Oct 30 03:48:48.640620 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:48.640593    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-cs62r"]
Oct 30 03:48:49.098945 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:49.098916    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:48:49.099134 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:49.099114    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="4fcc010f-9f67-4997-b009-c1330d4a01f4" containerName="otc-container" containerID="cri-o://ce6b86243c19b62761b644afe4e33c04d44f021af3c2c3bc0899339eb2ab2e00" gracePeriod=30
Oct 30 03:48:49.245995 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:49.245956    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:49.358468 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:49.358397    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:49.503000 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:49.502972    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ebf1223-c032-4175-bfad-07d478f9e689" path="/var/lib/kubelet/pods/0ebf1223-c032-4175-bfad-07d478f9e689/volumes"
Oct 30 03:48:50.119984 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.119946    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:48:50.135639 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:48:50.135602    4960 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[tempo-storage], unattached volumes=[tempo-storage], failed to process volumes=[]: context canceled" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8c3e78af-2fbd-4741-82df-e753ba7b2d2c"
Oct 30 03:48:50.188972 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.188915    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:50.251047 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.251018    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:50.282991 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.282962    4960 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:50.351661 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.351635    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:50.351847 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.351825    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:50.351961 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.351943    4960 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:50.367475 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.367453    4960 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") DevicePath \"csi-7de4f575c7745ab019b897e0ea854ec0444d04bc497b158bcde28fbb8cded20d\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:50.452576 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.452551    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-monitoring-tls-certs\") pod \"4fcc010f-9f67-4997-b009-c1330d4a01f4\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") "
Oct 30 03:48:50.452699 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.452671    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-tls-certs\") pod \"4fcc010f-9f67-4997-b009-c1330d4a01f4\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") "
Oct 30 03:48:50.452764 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.452722    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fcc010f-9f67-4997-b009-c1330d4a01f4-otc-internal\" (UniqueName: \"kubernetes.io/configmap/4fcc010f-9f67-4997-b009-c1330d4a01f4-otc-internal\") pod \"4fcc010f-9f67-4997-b009-c1330d4a01f4\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") "
Oct 30 03:48:50.452764 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.452752    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fcc010f-9f67-4997-b009-c1330d4a01f4-kube-api-access-j4zx6\" (UniqueName: \"kubernetes.io/projected/4fcc010f-9f67-4997-b009-c1330d4a01f4-kube-api-access-j4zx6\") pod \"4fcc010f-9f67-4997-b009-c1330d4a01f4\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") "
Oct 30 03:48:50.452855 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.452791    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-data-science-collector-ta-client-cert\") pod \"4fcc010f-9f67-4997-b009-c1330d4a01f4\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") "
Oct 30 03:48:50.452855 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.452841    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-otlp-tls-certs\") pod \"4fcc010f-9f67-4997-b009-c1330d4a01f4\" (UID: \"4fcc010f-9f67-4997-b009-c1330d4a01f4\") "
Oct 30 03:48:50.453142 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.453120    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fcc010f-9f67-4997-b009-c1330d4a01f4-otc-internal" pod "4fcc010f-9f67-4997-b009-c1330d4a01f4" (UID: "4fcc010f-9f67-4997-b009-c1330d4a01f4"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:50.454988 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.454958    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fcc010f-9f67-4997-b009-c1330d4a01f4-kube-api-access-j4zx6" pod "4fcc010f-9f67-4997-b009-c1330d4a01f4" (UID: "4fcc010f-9f67-4997-b009-c1330d4a01f4"). InnerVolumeSpecName "kube-api-access-j4zx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:48:50.455081 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.454999    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-monitoring-tls-certs" pod "4fcc010f-9f67-4997-b009-c1330d4a01f4" (UID: "4fcc010f-9f67-4997-b009-c1330d4a01f4"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:50.455626 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.455605    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-tls-certs" pod "4fcc010f-9f67-4997-b009-c1330d4a01f4" (UID: "4fcc010f-9f67-4997-b009-c1330d4a01f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:50.455710 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.455689    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-data-science-collector-ta-client-cert" pod "4fcc010f-9f67-4997-b009-c1330d4a01f4" (UID: "4fcc010f-9f67-4997-b009-c1330d4a01f4"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:50.455748 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.455731    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-otlp-tls-certs" pod "4fcc010f-9f67-4997-b009-c1330d4a01f4" (UID: "4fcc010f-9f67-4997-b009-c1330d4a01f4"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:50.471152 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.471123    4960 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/0f4ace3e6f98968798883337bc6fba9989453020fd940336937c79983332b77b/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:50.499308 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.499288    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:50.553545 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.553525    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:50.553545 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.553548    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4fcc010f-9f67-4997-b009-c1330d4a01f4-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:50.553724 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.553557    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4zx6\" (UniqueName: \"kubernetes.io/projected/4fcc010f-9f67-4997-b009-c1330d4a01f4-kube-api-access-j4zx6\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:50.553724 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.553568    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:50.553724 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.553576    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:50.553724 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.553585    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc010f-9f67-4997-b009-c1330d4a01f4-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:50.584598 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.584574    4960 generic.go:356] "Generic (PLEG): container finished" podID="4fcc010f-9f67-4997-b009-c1330d4a01f4" containerID="ce6b86243c19b62761b644afe4e33c04d44f021af3c2c3bc0899339eb2ab2e00" exitCode=0
Oct 30 03:48:50.584681 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.584617    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:50.584681 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.584662    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4fcc010f-9f67-4997-b009-c1330d4a01f4","Type":"ContainerDied","Data":"ce6b86243c19b62761b644afe4e33c04d44f021af3c2c3bc0899339eb2ab2e00"}
Oct 30 03:48:50.584765 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.584709    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4fcc010f-9f67-4997-b009-c1330d4a01f4","Type":"ContainerDied","Data":"9a3a4d94a21d85400d0321084f6369360d2fcc20a5a6c3e07c778d8b278bde00"}
Oct 30 03:48:50.584765 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.584734    4960 scope.go:122] "RemoveContainer" containerID="ce6b86243c19b62761b644afe4e33c04d44f021af3c2c3bc0899339eb2ab2e00"
Oct 30 03:48:50.584856 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.584838    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:50.593394 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.593375    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:50.593477 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.593398    4960 scope.go:122] "RemoveContainer" containerID="ce6b86243c19b62761b644afe4e33c04d44f021af3c2c3bc0899339eb2ab2e00"
Oct 30 03:48:50.593665 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:48:50.593649    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce6b86243c19b62761b644afe4e33c04d44f021af3c2c3bc0899339eb2ab2e00\": container with ID starting with ce6b86243c19b62761b644afe4e33c04d44f021af3c2c3bc0899339eb2ab2e00 not found: ID does not exist" containerID="ce6b86243c19b62761b644afe4e33c04d44f021af3c2c3bc0899339eb2ab2e00"
Oct 30 03:48:50.593706 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.593672    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce6b86243c19b62761b644afe4e33c04d44f021af3c2c3bc0899339eb2ab2e00"} err="failed to get container status \"ce6b86243c19b62761b644afe4e33c04d44f021af3c2c3bc0899339eb2ab2e00\": rpc error: code = NotFound desc = could not find container \"ce6b86243c19b62761b644afe4e33c04d44f021af3c2c3bc0899339eb2ab2e00\": container with ID starting with ce6b86243c19b62761b644afe4e33c04d44f021af3c2c3bc0899339eb2ab2e00 not found: ID does not exist"
Oct 30 03:48:50.681528 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.681505    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:48:50.685002 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.684978    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:48:50.755043 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.754983    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-gateway-tenants\") pod \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") "
Oct 30 03:48:50.755144 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.755111    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") "
Oct 30 03:48:50.755216 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.755161    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-mtls\") pod \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") "
Oct 30 03:48:50.755287 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.755246    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") "
Oct 30 03:48:50.755287 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.755275    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-conf\") pod \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") "
Oct 30 03:48:50.755459 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.755438    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") "
Oct 30 03:48:50.755545 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.755484    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") "
Oct 30 03:48:50.755625 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.755599    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-ca-bundle" pod "8c3e78af-2fbd-4741-82df-e753ba7b2d2c" (UID: "8c3e78af-2fbd-4741-82df-e753ba7b2d2c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:50.755815 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.755634    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-kube-api-access-kk8sm\" (UniqueName: \"kubernetes.io/projected/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-kube-api-access-kk8sm\") pod \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") "
Oct 30 03:48:50.755815 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.755685    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-conf" pod "8c3e78af-2fbd-4741-82df-e753ba7b2d2c" (UID: "8c3e78af-2fbd-4741-82df-e753ba7b2d2c"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:50.755815 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.755724    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-gateway-rbac\") pod \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\" (UID: \"8c3e78af-2fbd-4741-82df-e753ba7b2d2c\") "
Oct 30 03:48:50.756108 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.755981    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:50.756108 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.756010    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-conf\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:50.756255 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.756113    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-serving-cabundle" pod "8c3e78af-2fbd-4741-82df-e753ba7b2d2c" (UID: "8c3e78af-2fbd-4741-82df-e753ba7b2d2c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:50.757286 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.757248    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-gateway-rbac" pod "8c3e78af-2fbd-4741-82df-e753ba7b2d2c" (UID: "8c3e78af-2fbd-4741-82df-e753ba7b2d2c"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:50.757907 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.757881    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-gateway-tenants" pod "8c3e78af-2fbd-4741-82df-e753ba7b2d2c" (UID: "8c3e78af-2fbd-4741-82df-e753ba7b2d2c"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:50.758057 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.758034    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "8c3e78af-2fbd-4741-82df-e753ba7b2d2c" (UID: "8c3e78af-2fbd-4741-82df-e753ba7b2d2c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:50.758245 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.758216    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-mtls" pod "8c3e78af-2fbd-4741-82df-e753ba7b2d2c" (UID: "8c3e78af-2fbd-4741-82df-e753ba7b2d2c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:50.758245 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.758236    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-kube-api-access-kk8sm" pod "8c3e78af-2fbd-4741-82df-e753ba7b2d2c" (UID: "8c3e78af-2fbd-4741-82df-e753ba7b2d2c"). InnerVolumeSpecName "kube-api-access-kk8sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:48:50.758910 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.758895    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505" pod "8c3e78af-2fbd-4741-82df-e753ba7b2d2c" (UID: "8c3e78af-2fbd-4741-82df-e753ba7b2d2c"). InnerVolumeSpecName "pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:48:50.856647 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.856624    4960 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-gateway-rbac\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:50.856647 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.856644    4960 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-gateway-tenants\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:50.856828 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.856668    4960 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") on node \"ip-10-0-128-241.ec2.internal\" "
Oct 30 03:48:50.856828 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.856678    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:50.856828 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.856690    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:50.856828 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.856701    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:50.856828 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.856711    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk8sm\" (UniqueName: \"kubernetes.io/projected/8c3e78af-2fbd-4741-82df-e753ba7b2d2c-kube-api-access-kk8sm\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:48:50.868773 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.868758    4960 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505") on node "ip-10-0-128-241.ec2.internal"
Oct 30 03:48:50.957282 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:50.957245    4960 reconciler_common.go:299] "Volume detached for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"csi-7de4f575c7745ab019b897e0ea854ec0444d04bc497b158bcde28fbb8cded20d\""
Oct 30 03:48:51.502923 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:51.502852    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fcc010f-9f67-4997-b009-c1330d4a01f4" path="/var/lib/kubelet/pods/4fcc010f-9f67-4997-b009-c1330d4a01f4/volumes"
Oct 30 03:48:51.589880 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:51.589856    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:51.690715 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:51.690689    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:48:51.696745 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:51.696719    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:48:53.503683 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:53.503653    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c3e78af-2fbd-4741-82df-e753ba7b2d2c" path="/var/lib/kubelet/pods/8c3e78af-2fbd-4741-82df-e753ba7b2d2c/volumes"
Oct 30 03:48:53.953915 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:48:53.953878    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:48:54.012384 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.012358    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:54.255662 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.255592    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:48:54.255970 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.255957    4960 cpu_manager.go:414] "containerMap: removing container" podUID="0ebf1223-c032-4175-bfad-07d478f9e689" containerName="korrel8r"
Oct 30 03:48:54.256014 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.255971    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebf1223-c032-4175-bfad-07d478f9e689" containerName="korrel8r"
Oct 30 03:48:54.256014 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.255995    4960 cpu_manager.go:414] "containerMap: removing container" podUID="4fcc010f-9f67-4997-b009-c1330d4a01f4" containerName="otc-container"
Oct 30 03:48:54.256014 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.256000    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fcc010f-9f67-4997-b009-c1330d4a01f4" containerName="otc-container"
Oct 30 03:48:54.256100 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.256065    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebf1223-c032-4175-bfad-07d478f9e689" containerName="korrel8r"
Oct 30 03:48:54.256100 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.256074    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fcc010f-9f67-4997-b009-c1330d4a01f4" containerName="otc-container"
Oct 30 03:48:54.260767 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.260742    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.292060 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.292035    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:48:54.292209 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.292130    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wtzv9\""
Oct 30 03:48:54.292209 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.292194    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:48:54.293327 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.293308    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:48:54.305563 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.305546    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:48:54.305980 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.305963    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:48:54.306079 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.306034    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:48:54.383799 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.383778    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1a026bfe-a938-491c-af80-b2e4147141fa-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.383892 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.383808    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65pqd\" (UniqueName: \"kubernetes.io/projected/1a026bfe-a938-491c-af80-b2e4147141fa-kube-api-access-65pqd\") pod \"data-science-collector-collector-0\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.383892 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.383850    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.384023 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.383960    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.384023 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.384008    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.384102 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.384034    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.485397 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.485367    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.485503 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.485400    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.485503 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.485457    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1a026bfe-a938-491c-af80-b2e4147141fa-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.485503 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.485483    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65pqd\" (UniqueName: \"kubernetes.io/projected/1a026bfe-a938-491c-af80-b2e4147141fa-kube-api-access-65pqd\") pod \"data-science-collector-collector-0\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.485710 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.485687    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.485778 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.485739    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.486450 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.486422    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1a026bfe-a938-491c-af80-b2e4147141fa-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.488109 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.488088    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.488221 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.488169    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.488299 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.488252    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.488358 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.488330    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.505311 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.505286    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65pqd\" (UniqueName: \"kubernetes.io/projected/1a026bfe-a938-491c-af80-b2e4147141fa-kube-api-access-65pqd\") pod \"data-science-collector-collector-0\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.573112 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.573091    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.864039 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.864011    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:48:54.865837 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:48:54.865810    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a026bfe_a938_491c_af80_b2e4147141fa.slice/crio-aeacd64c46884388d00cf9f92c3e197b49c4eeb6df97434954c4dd50ec30f831 WatchSource:0}: Error finding container aeacd64c46884388d00cf9f92c3e197b49c4eeb6df97434954c4dd50ec30f831: Status 404 returned error can't find the container with id aeacd64c46884388d00cf9f92c3e197b49c4eeb6df97434954c4dd50ec30f831
Oct 30 03:48:54.867954 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.867922    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:48:54.868019 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:54.867984    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:48:55.614824 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:55.614790    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1a026bfe-a938-491c-af80-b2e4147141fa","Type":"ContainerStarted","Data":"dd3ee63c365cda89285916593ad0b9edcd806601244e54d91250d5c5bca87d8e"}
Oct 30 03:48:55.615209 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:55.614848    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1a026bfe-a938-491c-af80-b2e4147141fa","Type":"ContainerStarted","Data":"aeacd64c46884388d00cf9f92c3e197b49c4eeb6df97434954c4dd50ec30f831"}
Oct 30 03:48:55.658687 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:55.658647    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.658634861 podStartE2EDuration="1.658634861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:48:55.655727474 +0000 UTC m=+4016.304775894" watchObservedRunningTime="2026-10-30 03:48:55.658634861 +0000 UTC m=+4016.307683281"
Oct 30 03:48:56.553756 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:56.553704    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:56.667669 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:56.667630    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:56.786520 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:56.786489    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:56.869251 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:56.869222    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"]
Oct 30 03:48:56.872716 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:56.872698    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"
Oct 30 03:48:56.892942 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:56.892918    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"]
Oct 30 03:48:57.005878 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:57.005851    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4d5202fc-2dc4-4680-b3d5-5599377635d3-tls\") pod \"korrel8r-66d6f7f696-s4ppk\" (UID: \"4d5202fc-2dc4-4680-b3d5-5599377635d3\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"
Oct 30 03:48:57.006036 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:57.005923    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w6ml\" (UniqueName: \"kubernetes.io/projected/4d5202fc-2dc4-4680-b3d5-5599377635d3-kube-api-access-2w6ml\") pod \"korrel8r-66d6f7f696-s4ppk\" (UID: \"4d5202fc-2dc4-4680-b3d5-5599377635d3\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"
Oct 30 03:48:57.006123 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:57.006026    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d5202fc-2dc4-4680-b3d5-5599377635d3-config\") pod \"korrel8r-66d6f7f696-s4ppk\" (UID: \"4d5202fc-2dc4-4680-b3d5-5599377635d3\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"
Oct 30 03:48:57.106580 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:57.106552    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w6ml\" (UniqueName: \"kubernetes.io/projected/4d5202fc-2dc4-4680-b3d5-5599377635d3-kube-api-access-2w6ml\") pod \"korrel8r-66d6f7f696-s4ppk\" (UID: \"4d5202fc-2dc4-4680-b3d5-5599377635d3\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"
Oct 30 03:48:57.106752 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:57.106591    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d5202fc-2dc4-4680-b3d5-5599377635d3-config\") pod \"korrel8r-66d6f7f696-s4ppk\" (UID: \"4d5202fc-2dc4-4680-b3d5-5599377635d3\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"
Oct 30 03:48:57.106752 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:57.106642    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4d5202fc-2dc4-4680-b3d5-5599377635d3-tls\") pod \"korrel8r-66d6f7f696-s4ppk\" (UID: \"4d5202fc-2dc4-4680-b3d5-5599377635d3\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"
Oct 30 03:48:57.107472 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:57.107445    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d5202fc-2dc4-4680-b3d5-5599377635d3-config\") pod \"korrel8r-66d6f7f696-s4ppk\" (UID: \"4d5202fc-2dc4-4680-b3d5-5599377635d3\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"
Oct 30 03:48:57.109104 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:57.109080    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4d5202fc-2dc4-4680-b3d5-5599377635d3-tls\") pod \"korrel8r-66d6f7f696-s4ppk\" (UID: \"4d5202fc-2dc4-4680-b3d5-5599377635d3\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"
Oct 30 03:48:57.127329 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:57.127283    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w6ml\" (UniqueName: \"kubernetes.io/projected/4d5202fc-2dc4-4680-b3d5-5599377635d3-kube-api-access-2w6ml\") pod \"korrel8r-66d6f7f696-s4ppk\" (UID: \"4d5202fc-2dc4-4680-b3d5-5599377635d3\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"
Oct 30 03:48:57.169930 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:57.169906    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:57.185333 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:57.185312    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"
Oct 30 03:48:57.476847 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:57.476822    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"]
Oct 30 03:48:57.479757 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:48:57.479729    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5202fc_2dc4_4680_b3d5_5599377635d3.slice/crio-e6edb057f3bd1fc7a286a4103a7caf85a1b3268003dd3612db1fef386c6d8ec0 WatchSource:0}: Error finding container e6edb057f3bd1fc7a286a4103a7caf85a1b3268003dd3612db1fef386c6d8ec0: Status 404 returned error can't find the container with id e6edb057f3bd1fc7a286a4103a7caf85a1b3268003dd3612db1fef386c6d8ec0
Oct 30 03:48:57.626494 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:57.626463    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk" event={"ID":"4d5202fc-2dc4-4680-b3d5-5599377635d3","Type":"ContainerStarted","Data":"be3cf65d0a328874f06fe5f00dc0d10d0f590fa2b14ab06c845b3d5392548829"}
Oct 30 03:48:57.626626 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:57.626517    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk" event={"ID":"4d5202fc-2dc4-4680-b3d5-5599377635d3","Type":"ContainerStarted","Data":"e6edb057f3bd1fc7a286a4103a7caf85a1b3268003dd3612db1fef386c6d8ec0"}
Oct 30 03:48:57.626626 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:57.626572    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"
Oct 30 03:48:57.674460 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:57.674415    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk" podStartSLOduration=1.6744048550000001 podStartE2EDuration="1.674404855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:48:57.672400751 +0000 UTC m=+4018.321449187" watchObservedRunningTime="2026-10-30 03:48:57.674404855 +0000 UTC m=+4018.323453275"
Oct 30 03:48:58.458542 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:58.458510    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:58.570714 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:58.570679    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:58.783443 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:58.783364    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:59.867990 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:59.867953    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:48:59.872136 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:59.872115    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:48:59.888430 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:59.888407    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:48:59.888430 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:59.888421    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 03:48:59.901822 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:59.901756    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 03:48:59.901822 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:59.901799    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 03:48:59.902034 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:59.901940    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-tc6vt\""
Oct 30 03:48:59.902034 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:59.901947    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 03:48:59.902034 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:59.902003    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:48:59.902221 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:59.902086    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 03:48:59.903537 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:59.903515    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:48:59.945041 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:48:59.945007    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:00.030578 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.030554    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.030705 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.030584    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.030705 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.030611    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.030705 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.030682    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtn68\" (UniqueName: \"kubernetes.io/projected/db808cd0-f8fd-432e-a393-e5e3a092f365-kube-api-access-qtn68\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.030814 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.030751    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.030814 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.030801    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.030878 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.030854    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.030915 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.030895    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.131801 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.131746    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.131801 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.131776    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.131957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.131802    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.131957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.131825    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtn68\" (UniqueName: \"kubernetes.io/projected/db808cd0-f8fd-432e-a393-e5e3a092f365-kube-api-access-qtn68\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.131957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.131855    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.131957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.131872    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.131957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.131911    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.131957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.131931    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.132535 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.132512    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.132779 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.132759    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.132852 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.132763    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.132852 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.132763    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.134474 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.134455    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.134960 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.134939    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.134997 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.134948    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.152150 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.152121    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:00.152440 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.152423    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtn68\" (UniqueName: \"kubernetes.io/projected/db808cd0-f8fd-432e-a393-e5e3a092f365-kube-api-access-qtn68\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.232162 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.232131    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:49:00.232364 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.232336    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="1a026bfe-a938-491c-af80-b2e4147141fa" containerName="otc-container" containerID="cri-o://dd3ee63c365cda89285916593ad0b9edcd806601244e54d91250d5c5bca87d8e" gracePeriod=30
Oct 30 03:49:00.476357 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.476325    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:00.536106 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.535751    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.570723 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.570690    4960 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.636501 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.636476    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-otlp-tls-certs\") pod \"1a026bfe-a938-491c-af80-b2e4147141fa\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") "
Oct 30 03:49:00.636621 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.636525    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-tls-certs\") pod \"1a026bfe-a938-491c-af80-b2e4147141fa\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") "
Oct 30 03:49:00.636621 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.636592    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a026bfe-a938-491c-af80-b2e4147141fa-otc-internal\" (UniqueName: \"kubernetes.io/configmap/1a026bfe-a938-491c-af80-b2e4147141fa-otc-internal\") pod \"1a026bfe-a938-491c-af80-b2e4147141fa\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") "
Oct 30 03:49:00.636745 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.636632    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a026bfe-a938-491c-af80-b2e4147141fa-kube-api-access-65pqd\" (UniqueName: \"kubernetes.io/projected/1a026bfe-a938-491c-af80-b2e4147141fa-kube-api-access-65pqd\") pod \"1a026bfe-a938-491c-af80-b2e4147141fa\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") "
Oct 30 03:49:00.636745 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.636662    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-monitoring-tls-certs\") pod \"1a026bfe-a938-491c-af80-b2e4147141fa\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") "
Oct 30 03:49:00.636745 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.636716    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-data-science-collector-ta-client-cert\") pod \"1a026bfe-a938-491c-af80-b2e4147141fa\" (UID: \"1a026bfe-a938-491c-af80-b2e4147141fa\") "
Oct 30 03:49:00.636887 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.636870    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.637223 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.637019    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a026bfe-a938-491c-af80-b2e4147141fa-otc-internal" pod "1a026bfe-a938-491c-af80-b2e4147141fa" (UID: "1a026bfe-a938-491c-af80-b2e4147141fa"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:00.637223 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.637036    4960 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.639167 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.639135    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-otlp-tls-certs" pod "1a026bfe-a938-491c-af80-b2e4147141fa" (UID: "1a026bfe-a938-491c-af80-b2e4147141fa"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:00.639452 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.639425    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-tls-certs" pod "1a026bfe-a938-491c-af80-b2e4147141fa" (UID: "1a026bfe-a938-491c-af80-b2e4147141fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:00.639613 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.639590    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-monitoring-tls-certs" pod "1a026bfe-a938-491c-af80-b2e4147141fa" (UID: "1a026bfe-a938-491c-af80-b2e4147141fa"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:00.639613 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.639603    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-data-science-collector-ta-client-cert" pod "1a026bfe-a938-491c-af80-b2e4147141fa" (UID: "1a026bfe-a938-491c-af80-b2e4147141fa"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:00.639824 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.639799    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a026bfe-a938-491c-af80-b2e4147141fa-kube-api-access-65pqd" pod "1a026bfe-a938-491c-af80-b2e4147141fa" (UID: "1a026bfe-a938-491c-af80-b2e4147141fa"). InnerVolumeSpecName "kube-api-access-65pqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:00.643159 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.643129    4960 generic.go:356] "Generic (PLEG): container finished" podID="1a026bfe-a938-491c-af80-b2e4147141fa" containerID="dd3ee63c365cda89285916593ad0b9edcd806601244e54d91250d5c5bca87d8e" exitCode=0
Oct 30 03:49:00.643261 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.643172    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:00.643261 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.643217    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1a026bfe-a938-491c-af80-b2e4147141fa","Type":"ContainerDied","Data":"dd3ee63c365cda89285916593ad0b9edcd806601244e54d91250d5c5bca87d8e"}
Oct 30 03:49:00.643343 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.643265    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1a026bfe-a938-491c-af80-b2e4147141fa","Type":"ContainerDied","Data":"aeacd64c46884388d00cf9f92c3e197b49c4eeb6df97434954c4dd50ec30f831"}
Oct 30 03:49:00.643343 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.643286    4960 scope.go:122] "RemoveContainer" containerID="dd3ee63c365cda89285916593ad0b9edcd806601244e54d91250d5c5bca87d8e"
Oct 30 03:49:00.651703 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.651685    4960 scope.go:122] "RemoveContainer" containerID="dd3ee63c365cda89285916593ad0b9edcd806601244e54d91250d5c5bca87d8e"
Oct 30 03:49:00.651962 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:00.651945    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd3ee63c365cda89285916593ad0b9edcd806601244e54d91250d5c5bca87d8e\": container with ID starting with dd3ee63c365cda89285916593ad0b9edcd806601244e54d91250d5c5bca87d8e not found: ID does not exist" containerID="dd3ee63c365cda89285916593ad0b9edcd806601244e54d91250d5c5bca87d8e"
Oct 30 03:49:00.652007 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.651971    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd3ee63c365cda89285916593ad0b9edcd806601244e54d91250d5c5bca87d8e"} err="failed to get container status \"dd3ee63c365cda89285916593ad0b9edcd806601244e54d91250d5c5bca87d8e\": rpc error: code = NotFound desc = could not find container \"dd3ee63c365cda89285916593ad0b9edcd806601244e54d91250d5c5bca87d8e\": container with ID starting with dd3ee63c365cda89285916593ad0b9edcd806601244e54d91250d5c5bca87d8e not found: ID does not exist"
Oct 30 03:49:00.652696 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.652680    4960 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") DevicePath \"csi-7de4f575c7745ab019b897e0ea854ec0444d04bc497b158bcde28fbb8cded20d\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.697975 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.697953    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:49:00.707067 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.703345    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:49:00.737704 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.737681    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:00.737809 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.737709    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:00.737809 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.737723    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1a026bfe-a938-491c-af80-b2e4147141fa-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:00.737809 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.737736    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65pqd\" (UniqueName: \"kubernetes.io/projected/1a026bfe-a938-491c-af80-b2e4147141fa-kube-api-access-65pqd\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:00.737809 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.737749    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:00.737809 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.737765    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1a026bfe-a938-491c-af80-b2e4147141fa-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:00.753372 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.753353    4960 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/0f4ace3e6f98968798883337bc6fba9989453020fd940336937c79983332b77b/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:00.782035 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:00.782015    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:01.090445 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:01.090418    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:01.408372 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:01.408345    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:49:01.409971 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:49:01.409929    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb808cd0_f8fd_432e_a393_e5e3a092f365.slice/crio-e1732c05a895ae501c5c7851dc3cd4dc989f586daf1b2cbe82220c5119e38b28 WatchSource:0}: Error finding container e1732c05a895ae501c5c7851dc3cd4dc989f586daf1b2cbe82220c5119e38b28: Status 404 returned error can't find the container with id e1732c05a895ae501c5c7851dc3cd4dc989f586daf1b2cbe82220c5119e38b28
Oct 30 03:49:01.502638 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:01.502610    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a026bfe-a938-491c-af80-b2e4147141fa" path="/var/lib/kubelet/pods/1a026bfe-a938-491c-af80-b2e4147141fa/volumes"
Oct 30 03:49:01.651106 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:01.651070    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"db808cd0-f8fd-432e-a393-e5e3a092f365","Type":"ContainerStarted","Data":"28ef9955d6478a9006718eebf5dbf121b3ce6a72f128cc4b987d1cdc3ef8678b"}
Oct 30 03:49:01.651233 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:01.651124    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"db808cd0-f8fd-432e-a393-e5e3a092f365","Type":"ContainerStarted","Data":"50f5ef8aed395bfe9d71ca72335a173eb1ffcefb008ab13cb52b70aa9ff1bf19"}
Oct 30 03:49:01.651233 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:01.651141    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"db808cd0-f8fd-432e-a393-e5e3a092f365","Type":"ContainerStarted","Data":"dced237dc7abfb496e1b5e3b8aba176071fc830e6289452a9710fcab74fce969"}
Oct 30 03:49:01.651233 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:01.651155    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"db808cd0-f8fd-432e-a393-e5e3a092f365","Type":"ContainerStarted","Data":"e1732c05a895ae501c5c7851dc3cd4dc989f586daf1b2cbe82220c5119e38b28"}
Oct 30 03:49:01.651361 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:01.651273    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:01.653682 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:01.653667    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:01.698128 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:01.698055    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=2.698040408 podStartE2EDuration="2.698040408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:01.697368408 +0000 UTC m=+4022.346416823" watchObservedRunningTime="2026-10-30 03:49:01.698040408 +0000 UTC m=+4022.347088901"
Oct 30 03:49:02.656852 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:02.656820    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:02.657298 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:02.656886    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:02.664848 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:02.664826    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:05.321668 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:05.321637    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:05.522190 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:05.522145    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:05.630368 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:05.630297    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:05.697342 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:05.697316    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"]
Oct 30 03:49:05.697513 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:05.697494    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk" podUID="4d5202fc-2dc4-4680-b3d5-5599377635d3" containerName="korrel8r" containerID="cri-o://be3cf65d0a328874f06fe5f00dc0d10d0f590fa2b14ab06c845b3d5392548829" gracePeriod=30
Oct 30 03:49:05.699426 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:05.699402    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"
Oct 30 03:49:05.947072 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:05.947049    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"
Oct 30 03:49:06.081931 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:06.081904    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d5202fc-2dc4-4680-b3d5-5599377635d3-kube-api-access-2w6ml\" (UniqueName: \"kubernetes.io/projected/4d5202fc-2dc4-4680-b3d5-5599377635d3-kube-api-access-2w6ml\") pod \"4d5202fc-2dc4-4680-b3d5-5599377635d3\" (UID: \"4d5202fc-2dc4-4680-b3d5-5599377635d3\") "
Oct 30 03:49:06.082087 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:06.081941    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d5202fc-2dc4-4680-b3d5-5599377635d3-tls\" (UniqueName: \"kubernetes.io/secret/4d5202fc-2dc4-4680-b3d5-5599377635d3-tls\") pod \"4d5202fc-2dc4-4680-b3d5-5599377635d3\" (UID: \"4d5202fc-2dc4-4680-b3d5-5599377635d3\") "
Oct 30 03:49:06.082087 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:06.081984    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d5202fc-2dc4-4680-b3d5-5599377635d3-config\" (UniqueName: \"kubernetes.io/configmap/4d5202fc-2dc4-4680-b3d5-5599377635d3-config\") pod \"4d5202fc-2dc4-4680-b3d5-5599377635d3\" (UID: \"4d5202fc-2dc4-4680-b3d5-5599377635d3\") "
Oct 30 03:49:06.082427 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:06.082408    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d5202fc-2dc4-4680-b3d5-5599377635d3-config" pod "4d5202fc-2dc4-4680-b3d5-5599377635d3" (UID: "4d5202fc-2dc4-4680-b3d5-5599377635d3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:06.084097 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:06.084070    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5202fc-2dc4-4680-b3d5-5599377635d3-kube-api-access-2w6ml" pod "4d5202fc-2dc4-4680-b3d5-5599377635d3" (UID: "4d5202fc-2dc4-4680-b3d5-5599377635d3"). InnerVolumeSpecName "kube-api-access-2w6ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:06.084216 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:06.084113    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d5202fc-2dc4-4680-b3d5-5599377635d3-tls" pod "4d5202fc-2dc4-4680-b3d5-5599377635d3" (UID: "4d5202fc-2dc4-4680-b3d5-5599377635d3"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:06.183475 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:06.183411    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w6ml\" (UniqueName: \"kubernetes.io/projected/4d5202fc-2dc4-4680-b3d5-5599377635d3-kube-api-access-2w6ml\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:06.183475 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:06.183442    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4d5202fc-2dc4-4680-b3d5-5599377635d3-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:06.183475 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:06.183451    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d5202fc-2dc4-4680-b3d5-5599377635d3-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:06.675336 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:06.675307    4960 generic.go:356] "Generic (PLEG): container finished" podID="4d5202fc-2dc4-4680-b3d5-5599377635d3" containerID="be3cf65d0a328874f06fe5f00dc0d10d0f590fa2b14ab06c845b3d5392548829" exitCode=0
Oct 30 03:49:06.675756 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:06.675356    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"
Oct 30 03:49:06.675756 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:06.675398    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk" event={"ID":"4d5202fc-2dc4-4680-b3d5-5599377635d3","Type":"ContainerDied","Data":"be3cf65d0a328874f06fe5f00dc0d10d0f590fa2b14ab06c845b3d5392548829"}
Oct 30 03:49:06.675756 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:06.675445    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk" event={"ID":"4d5202fc-2dc4-4680-b3d5-5599377635d3","Type":"ContainerDied","Data":"e6edb057f3bd1fc7a286a4103a7caf85a1b3268003dd3612db1fef386c6d8ec0"}
Oct 30 03:49:06.675756 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:06.675466    4960 scope.go:122] "RemoveContainer" containerID="be3cf65d0a328874f06fe5f00dc0d10d0f590fa2b14ab06c845b3d5392548829"
Oct 30 03:49:06.684619 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:06.684599    4960 scope.go:122] "RemoveContainer" containerID="be3cf65d0a328874f06fe5f00dc0d10d0f590fa2b14ab06c845b3d5392548829"
Oct 30 03:49:06.684881 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:06.684864    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3cf65d0a328874f06fe5f00dc0d10d0f590fa2b14ab06c845b3d5392548829\": container with ID starting with be3cf65d0a328874f06fe5f00dc0d10d0f590fa2b14ab06c845b3d5392548829 not found: ID does not exist" containerID="be3cf65d0a328874f06fe5f00dc0d10d0f590fa2b14ab06c845b3d5392548829"
Oct 30 03:49:06.684935 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:06.684889    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3cf65d0a328874f06fe5f00dc0d10d0f590fa2b14ab06c845b3d5392548829"} err="failed to get container status \"be3cf65d0a328874f06fe5f00dc0d10d0f590fa2b14ab06c845b3d5392548829\": rpc error: code = NotFound desc = could not find container \"be3cf65d0a328874f06fe5f00dc0d10d0f590fa2b14ab06c845b3d5392548829\": container with ID starting with be3cf65d0a328874f06fe5f00dc0d10d0f590fa2b14ab06c845b3d5392548829 not found: ID does not exist"
Oct 30 03:49:06.732793 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:06.730966    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"]
Oct 30 03:49:06.743311 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:06.743290    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-s4ppk"]
Oct 30 03:49:07.233724 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:07.233687    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:07.344628 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:07.344597    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:07.502453 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:07.502382    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d5202fc-2dc4-4680-b3d5-5599377635d3" path="/var/lib/kubelet/pods/4d5202fc-2dc4-4680-b3d5-5599377635d3/volumes"
Oct 30 03:49:08.106728 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:08.106690    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:49:08.107245 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:08.106961    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="db808cd0-f8fd-432e-a393-e5e3a092f365" containerName="tempo" containerID="cri-o://dced237dc7abfb496e1b5e3b8aba176071fc830e6289452a9710fcab74fce969" gracePeriod=30
Oct 30 03:49:08.107417 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:08.107379    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="db808cd0-f8fd-432e-a393-e5e3a092f365" containerName="tempo-gateway-opa" containerID="cri-o://28ef9955d6478a9006718eebf5dbf121b3ce6a72f128cc4b987d1cdc3ef8678b" gracePeriod=30
Oct 30 03:49:08.108011 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:08.107393    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="db808cd0-f8fd-432e-a393-e5e3a092f365" containerName="tempo-gateway" containerID="cri-o://50f5ef8aed395bfe9d71ca72335a173eb1ffcefb008ab13cb52b70aa9ff1bf19" gracePeriod=30
Oct 30 03:49:08.205692 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:08.205659    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:08.687259 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:08.687227    4960 generic.go:356] "Generic (PLEG): container finished" podID="db808cd0-f8fd-432e-a393-e5e3a092f365" containerID="28ef9955d6478a9006718eebf5dbf121b3ce6a72f128cc4b987d1cdc3ef8678b" exitCode=0
Oct 30 03:49:08.687259 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:08.687252    4960 generic.go:356] "Generic (PLEG): container finished" podID="db808cd0-f8fd-432e-a393-e5e3a092f365" containerID="dced237dc7abfb496e1b5e3b8aba176071fc830e6289452a9710fcab74fce969" exitCode=0
Oct 30 03:49:08.687459 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:08.687300    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"db808cd0-f8fd-432e-a393-e5e3a092f365","Type":"ContainerDied","Data":"28ef9955d6478a9006718eebf5dbf121b3ce6a72f128cc4b987d1cdc3ef8678b"}
Oct 30 03:49:08.687459 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:08.687347    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"db808cd0-f8fd-432e-a393-e5e3a092f365","Type":"ContainerDied","Data":"dced237dc7abfb496e1b5e3b8aba176071fc830e6289452a9710fcab74fce969"}
Oct 30 03:49:09.389886 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.389863    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:09.510767 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.510687    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-ca-bundle\") pod \"db808cd0-f8fd-432e-a393-e5e3a092f365\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") "
Oct 30 03:49:09.510767 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.510749    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db808cd0-f8fd-432e-a393-e5e3a092f365-kube-api-access-qtn68\" (UniqueName: \"kubernetes.io/projected/db808cd0-f8fd-432e-a393-e5e3a092f365-kube-api-access-qtn68\") pod \"db808cd0-f8fd-432e-a393-e5e3a092f365\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") "
Oct 30 03:49:09.510991 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.510784    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-gateway-tenants\") pod \"db808cd0-f8fd-432e-a393-e5e3a092f365\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") "
Oct 30 03:49:09.510991 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.510802    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-gateway-rbac\") pod \"db808cd0-f8fd-432e-a393-e5e3a092f365\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") "
Oct 30 03:49:09.510991 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.510831    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-conf\") pod \"db808cd0-f8fd-432e-a393-e5e3a092f365\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") "
Oct 30 03:49:09.510991 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.510856    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"db808cd0-f8fd-432e-a393-e5e3a092f365\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") "
Oct 30 03:49:09.510991 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.510914    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-mtls\") pod \"db808cd0-f8fd-432e-a393-e5e3a092f365\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") "
Oct 30 03:49:09.510991 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.510954    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"db808cd0-f8fd-432e-a393-e5e3a092f365\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") "
Oct 30 03:49:09.511365 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.511056    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"db808cd0-f8fd-432e-a393-e5e3a092f365\" (UID: \"db808cd0-f8fd-432e-a393-e5e3a092f365\") "
Oct 30 03:49:09.511365 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.511094    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-ca-bundle" pod "db808cd0-f8fd-432e-a393-e5e3a092f365" (UID: "db808cd0-f8fd-432e-a393-e5e3a092f365"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:09.511365 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.511231    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-gateway-rbac" pod "db808cd0-f8fd-432e-a393-e5e3a092f365" (UID: "db808cd0-f8fd-432e-a393-e5e3a092f365"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:09.511683 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.511457    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:09.511683 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.511477    4960 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-gateway-rbac\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:09.511683 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.511504    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-conf" pod "db808cd0-f8fd-432e-a393-e5e3a092f365" (UID: "db808cd0-f8fd-432e-a393-e5e3a092f365"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:09.511683 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.511581    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-serving-cabundle" pod "db808cd0-f8fd-432e-a393-e5e3a092f365" (UID: "db808cd0-f8fd-432e-a393-e5e3a092f365"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:09.513339 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.513298    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db808cd0-f8fd-432e-a393-e5e3a092f365-kube-api-access-qtn68" pod "db808cd0-f8fd-432e-a393-e5e3a092f365" (UID: "db808cd0-f8fd-432e-a393-e5e3a092f365"). InnerVolumeSpecName "kube-api-access-qtn68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:09.513686 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.513658    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "db808cd0-f8fd-432e-a393-e5e3a092f365" (UID: "db808cd0-f8fd-432e-a393-e5e3a092f365"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:09.513782 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.513720    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-gateway-tenants" pod "db808cd0-f8fd-432e-a393-e5e3a092f365" (UID: "db808cd0-f8fd-432e-a393-e5e3a092f365"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:09.513782 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.513737    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-mtls" pod "db808cd0-f8fd-432e-a393-e5e3a092f365" (UID: "db808cd0-f8fd-432e-a393-e5e3a092f365"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:09.514157 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.514138    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505" pod "db808cd0-f8fd-432e-a393-e5e3a092f365" (UID: "db808cd0-f8fd-432e-a393-e5e3a092f365"). InnerVolumeSpecName "pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:49:09.612404 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.612368    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:09.612404 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.612399    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:09.612640 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.612436    4960 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") on node \"ip-10-0-128-241.ec2.internal\" "
Oct 30 03:49:09.612640 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.612449    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtn68\" (UniqueName: \"kubernetes.io/projected/db808cd0-f8fd-432e-a393-e5e3a092f365-kube-api-access-qtn68\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:09.612640 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.612462    4960 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-gateway-tenants\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:09.612640 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.612474    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-conf\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:09.612640 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.612486    4960 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/db808cd0-f8fd-432e-a393-e5e3a092f365-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:09.622675 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.622657    4960 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505") on node "ip-10-0-128-241.ec2.internal"
Oct 30 03:49:09.694116 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.694087    4960 generic.go:356] "Generic (PLEG): container finished" podID="db808cd0-f8fd-432e-a393-e5e3a092f365" containerID="50f5ef8aed395bfe9d71ca72335a173eb1ffcefb008ab13cb52b70aa9ff1bf19" exitCode=0
Oct 30 03:49:09.694255 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.694142    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:49:09.694255 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.694208    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"db808cd0-f8fd-432e-a393-e5e3a092f365","Type":"ContainerDied","Data":"50f5ef8aed395bfe9d71ca72335a173eb1ffcefb008ab13cb52b70aa9ff1bf19"}
Oct 30 03:49:09.694359 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.694268    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"db808cd0-f8fd-432e-a393-e5e3a092f365","Type":"ContainerDied","Data":"e1732c05a895ae501c5c7851dc3cd4dc989f586daf1b2cbe82220c5119e38b28"}
Oct 30 03:49:09.694359 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.694295    4960 scope.go:122] "RemoveContainer" containerID="28ef9955d6478a9006718eebf5dbf121b3ce6a72f128cc4b987d1cdc3ef8678b"
Oct 30 03:49:09.703288 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.703272    4960 scope.go:122] "RemoveContainer" containerID="50f5ef8aed395bfe9d71ca72335a173eb1ffcefb008ab13cb52b70aa9ff1bf19"
Oct 30 03:49:09.711391 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.711376    4960 scope.go:122] "RemoveContainer" containerID="dced237dc7abfb496e1b5e3b8aba176071fc830e6289452a9710fcab74fce969"
Oct 30 03:49:09.713568 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.713543    4960 reconciler_common.go:299] "Volume detached for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"csi-7de4f575c7745ab019b897e0ea854ec0444d04bc497b158bcde28fbb8cded20d\""
Oct 30 03:49:09.718909 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.718892    4960 scope.go:122] "RemoveContainer" containerID="28ef9955d6478a9006718eebf5dbf121b3ce6a72f128cc4b987d1cdc3ef8678b"
Oct 30 03:49:09.719154 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:09.719137    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ef9955d6478a9006718eebf5dbf121b3ce6a72f128cc4b987d1cdc3ef8678b\": container with ID starting with 28ef9955d6478a9006718eebf5dbf121b3ce6a72f128cc4b987d1cdc3ef8678b not found: ID does not exist" containerID="28ef9955d6478a9006718eebf5dbf121b3ce6a72f128cc4b987d1cdc3ef8678b"
Oct 30 03:49:09.719252 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.719161    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ef9955d6478a9006718eebf5dbf121b3ce6a72f128cc4b987d1cdc3ef8678b"} err="failed to get container status \"28ef9955d6478a9006718eebf5dbf121b3ce6a72f128cc4b987d1cdc3ef8678b\": rpc error: code = NotFound desc = could not find container \"28ef9955d6478a9006718eebf5dbf121b3ce6a72f128cc4b987d1cdc3ef8678b\": container with ID starting with 28ef9955d6478a9006718eebf5dbf121b3ce6a72f128cc4b987d1cdc3ef8678b not found: ID does not exist"
Oct 30 03:49:09.719252 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.719191    4960 scope.go:122] "RemoveContainer" containerID="50f5ef8aed395bfe9d71ca72335a173eb1ffcefb008ab13cb52b70aa9ff1bf19"
Oct 30 03:49:09.719469 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:09.719441    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50f5ef8aed395bfe9d71ca72335a173eb1ffcefb008ab13cb52b70aa9ff1bf19\": container with ID starting with 50f5ef8aed395bfe9d71ca72335a173eb1ffcefb008ab13cb52b70aa9ff1bf19 not found: ID does not exist" containerID="50f5ef8aed395bfe9d71ca72335a173eb1ffcefb008ab13cb52b70aa9ff1bf19"
Oct 30 03:49:09.719510 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.719475    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f5ef8aed395bfe9d71ca72335a173eb1ffcefb008ab13cb52b70aa9ff1bf19"} err="failed to get container status \"50f5ef8aed395bfe9d71ca72335a173eb1ffcefb008ab13cb52b70aa9ff1bf19\": rpc error: code = NotFound desc = could not find container \"50f5ef8aed395bfe9d71ca72335a173eb1ffcefb008ab13cb52b70aa9ff1bf19\": container with ID starting with 50f5ef8aed395bfe9d71ca72335a173eb1ffcefb008ab13cb52b70aa9ff1bf19 not found: ID does not exist"
Oct 30 03:49:09.719510 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.719490    4960 scope.go:122] "RemoveContainer" containerID="dced237dc7abfb496e1b5e3b8aba176071fc830e6289452a9710fcab74fce969"
Oct 30 03:49:09.719716 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:09.719698    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dced237dc7abfb496e1b5e3b8aba176071fc830e6289452a9710fcab74fce969\": container with ID starting with dced237dc7abfb496e1b5e3b8aba176071fc830e6289452a9710fcab74fce969 not found: ID does not exist" containerID="dced237dc7abfb496e1b5e3b8aba176071fc830e6289452a9710fcab74fce969"
Oct 30 03:49:09.719756 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.719721    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dced237dc7abfb496e1b5e3b8aba176071fc830e6289452a9710fcab74fce969"} err="failed to get container status \"dced237dc7abfb496e1b5e3b8aba176071fc830e6289452a9710fcab74fce969\": rpc error: code = NotFound desc = could not find container \"dced237dc7abfb496e1b5e3b8aba176071fc830e6289452a9710fcab74fce969\": container with ID starting with dced237dc7abfb496e1b5e3b8aba176071fc830e6289452a9710fcab74fce969 not found: ID does not exist"
Oct 30 03:49:09.753836 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.753780    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:49:09.758436 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:09.758417    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:49:11.283559 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.283526    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:49:11.283957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.283861    4960 cpu_manager.go:414] "containerMap: removing container" podUID="1a026bfe-a938-491c-af80-b2e4147141fa" containerName="otc-container"
Oct 30 03:49:11.283957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.283871    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a026bfe-a938-491c-af80-b2e4147141fa" containerName="otc-container"
Oct 30 03:49:11.283957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.283880    4960 cpu_manager.go:414] "containerMap: removing container" podUID="db808cd0-f8fd-432e-a393-e5e3a092f365" containerName="tempo"
Oct 30 03:49:11.283957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.283887    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db808cd0-f8fd-432e-a393-e5e3a092f365" containerName="tempo"
Oct 30 03:49:11.283957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.283913    4960 cpu_manager.go:414] "containerMap: removing container" podUID="db808cd0-f8fd-432e-a393-e5e3a092f365" containerName="tempo-gateway"
Oct 30 03:49:11.283957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.283919    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db808cd0-f8fd-432e-a393-e5e3a092f365" containerName="tempo-gateway"
Oct 30 03:49:11.283957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.283925    4960 cpu_manager.go:414] "containerMap: removing container" podUID="db808cd0-f8fd-432e-a393-e5e3a092f365" containerName="tempo-gateway-opa"
Oct 30 03:49:11.283957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.283929    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db808cd0-f8fd-432e-a393-e5e3a092f365" containerName="tempo-gateway-opa"
Oct 30 03:49:11.283957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.283937    4960 cpu_manager.go:414] "containerMap: removing container" podUID="4d5202fc-2dc4-4680-b3d5-5599377635d3" containerName="korrel8r"
Oct 30 03:49:11.283957 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.283941    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d5202fc-2dc4-4680-b3d5-5599377635d3" containerName="korrel8r"
Oct 30 03:49:11.284315 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.284008    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a026bfe-a938-491c-af80-b2e4147141fa" containerName="otc-container"
Oct 30 03:49:11.284315 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.284015    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="db808cd0-f8fd-432e-a393-e5e3a092f365" containerName="tempo"
Oct 30 03:49:11.284315 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.284024    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="db808cd0-f8fd-432e-a393-e5e3a092f365" containerName="tempo-gateway"
Oct 30 03:49:11.284315 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.284030    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="db808cd0-f8fd-432e-a393-e5e3a092f365" containerName="tempo-gateway-opa"
Oct 30 03:49:11.284315 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.284037    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d5202fc-2dc4-4680-b3d5-5599377635d3" containerName="korrel8r"
Oct 30 03:49:11.288566 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.288547    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.306958 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.306933    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:49:11.307088 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.307047    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:49:11.308336 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.308314    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wtzv9\""
Oct 30 03:49:11.308600 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.308582    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:49:11.319452 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.319432    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:49:11.319546 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.319479    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:49:11.319546 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.319513    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:49:11.428861 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.428834    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.429027 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.428877    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pj92\" (UniqueName: \"kubernetes.io/projected/a73780b3-6931-44ea-9a04-969fe86f9d25-kube-api-access-8pj92\") pod \"data-science-collector-collector-0\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.429027 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.428983    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.429144 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.429050    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a73780b3-6931-44ea-9a04-969fe86f9d25-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.429225 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.429160    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.429279 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.429217    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.512697 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.512666    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db808cd0-f8fd-432e-a393-e5e3a092f365" path="/var/lib/kubelet/pods/db808cd0-f8fd-432e-a393-e5e3a092f365/volumes"
Oct 30 03:49:11.529752 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.529721    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a73780b3-6931-44ea-9a04-969fe86f9d25-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.529923 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.529758    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.529923 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.529778    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.529923 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.529805    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.529923 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.529850    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pj92\" (UniqueName: \"kubernetes.io/projected/a73780b3-6931-44ea-9a04-969fe86f9d25-kube-api-access-8pj92\") pod \"data-science-collector-collector-0\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.529923 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.529906    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.530654 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.530554    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a73780b3-6931-44ea-9a04-969fe86f9d25-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.532477 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.532454    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.532576 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.532553    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.532668 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.532651    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.532703 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.532663    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.550622 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.550571    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pj92\" (UniqueName: \"kubernetes.io/projected/a73780b3-6931-44ea-9a04-969fe86f9d25-kube-api-access-8pj92\") pod \"data-science-collector-collector-0\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.601467 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.601446    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:11.737768 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.737734    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"]
Oct 30 03:49:11.737963 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.737941    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk" podUID="6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb" containerName="korrel8r" containerID="cri-o://29e404c7f2875c58d8d2c346a092aba0c06abab63dfbd37ed1eb55d12a3c2c59" gracePeriod=30
Oct 30 03:49:11.899612 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.899586    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:49:11.901536 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:49:11.901509    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73780b3_6931_44ea_9a04_969fe86f9d25.slice/crio-bc3d73723da20d9f42a6b9bfb870fcd7f583c8a275e52269aa79c217f0a0a956 WatchSource:0}: Error finding container bc3d73723da20d9f42a6b9bfb870fcd7f583c8a275e52269aa79c217f0a0a956: Status 404 returned error can't find the container with id bc3d73723da20d9f42a6b9bfb870fcd7f583c8a275e52269aa79c217f0a0a956
Oct 30 03:49:11.903534 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.903502    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:49:11.903642 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.903568    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:49:11.978951 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:11.978924    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"
Oct 30 03:49:12.035335 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.034757    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-kube-api-access-422mq\" (UniqueName: \"kubernetes.io/projected/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-kube-api-access-422mq\") pod \"6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb\" (UID: \"6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb\") "
Oct 30 03:49:12.035335 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.034812    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-tls\" (UniqueName: \"kubernetes.io/secret/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-tls\") pod \"6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb\" (UID: \"6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb\") "
Oct 30 03:49:12.035335 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.034939    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-config\" (UniqueName: \"kubernetes.io/configmap/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-config\") pod \"6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb\" (UID: \"6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb\") "
Oct 30 03:49:12.035716 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.035675    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-config" pod "6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb" (UID: "6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:12.038421 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.038392    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-tls" pod "6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb" (UID: "6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:12.039914 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.039866    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-kube-api-access-422mq" pod "6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb" (UID: "6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb"). InnerVolumeSpecName "kube-api-access-422mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:12.119345 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.119262    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:12.136793 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.136764    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:12.136793 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.136792    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:12.137001 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.136808    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-422mq\" (UniqueName: \"kubernetes.io/projected/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb-kube-api-access-422mq\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:12.186844 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.186079    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 03:49:12.186844 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.186392    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="33a822a2-33ea-4a2c-93e8-383364e7593d" containerName="alertmanager" containerID="cri-o://36ee9f2f0cf8f33f35092439422ee951d08c265842ffc2569646213a3a48a152" gracePeriod=120
Oct 30 03:49:12.186844 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.186560    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="33a822a2-33ea-4a2c-93e8-383364e7593d" containerName="config-reloader" containerID="cri-o://1ae073d5f0e640139548e908ef7b4c712486362710b8a896508596acb7116215" gracePeriod=120
Oct 30 03:49:12.193651 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.193441    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:49:12.194807 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.194135    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="fb57c6c2-5791-478a-bfa2-c98fedc041ed" containerName="prometheus" containerID="cri-o://c53458936c2a39d594173225ff5151c76a44e407fa9e425c80dcef158bb6b9c2" gracePeriod=600
Oct 30 03:49:12.194807 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.194159    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="fb57c6c2-5791-478a-bfa2-c98fedc041ed" containerName="thanos-sidecar" containerID="cri-o://cf2b5d51ee3a9449fd8f7f34bf24de2fcc39c09d6bd5468fc8590d863c5d4673" gracePeriod=600
Oct 30 03:49:12.194807 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.194258    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="fb57c6c2-5791-478a-bfa2-c98fedc041ed" containerName="config-reloader" containerID="cri-o://4d3c374d00c1fea545c169e50f690bd7195dc60ef189082375d4841c328fe6d4" gracePeriod=600
Oct 30 03:49:12.237929 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.237898    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:12.356548 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.356514    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:12.521400 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.521373    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:12.629540 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.629506    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:12.709866 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.709795    4960 generic.go:356] "Generic (PLEG): container finished" podID="6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb" containerID="29e404c7f2875c58d8d2c346a092aba0c06abab63dfbd37ed1eb55d12a3c2c59" exitCode=0
Oct 30 03:49:12.709866 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.709834    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"
Oct 30 03:49:12.710043 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.709879    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk" event={"ID":"6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb","Type":"ContainerDied","Data":"29e404c7f2875c58d8d2c346a092aba0c06abab63dfbd37ed1eb55d12a3c2c59"}
Oct 30 03:49:12.710043 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.709926    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk" event={"ID":"6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb","Type":"ContainerDied","Data":"15298efc949cbddd867a6fedff9e8747a7ee00001b9cc3da8084b78c6a0792f8"}
Oct 30 03:49:12.710043 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.709957    4960 scope.go:122] "RemoveContainer" containerID="29e404c7f2875c58d8d2c346a092aba0c06abab63dfbd37ed1eb55d12a3c2c59"
Oct 30 03:49:12.711748 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.711725    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a73780b3-6931-44ea-9a04-969fe86f9d25","Type":"ContainerStarted","Data":"6e8f15f363828ec855b11e0fafa67be66f53ec01dc5178595371be9768e9a785"}
Oct 30 03:49:12.711837 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.711771    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a73780b3-6931-44ea-9a04-969fe86f9d25","Type":"ContainerStarted","Data":"bc3d73723da20d9f42a6b9bfb870fcd7f583c8a275e52269aa79c217f0a0a956"}
Oct 30 03:49:12.714313 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.714288    4960 generic.go:356] "Generic (PLEG): container finished" podID="33a822a2-33ea-4a2c-93e8-383364e7593d" containerID="1ae073d5f0e640139548e908ef7b4c712486362710b8a896508596acb7116215" exitCode=0
Oct 30 03:49:12.714440 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.714365    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"33a822a2-33ea-4a2c-93e8-383364e7593d","Type":"ContainerDied","Data":"1ae073d5f0e640139548e908ef7b4c712486362710b8a896508596acb7116215"}
Oct 30 03:49:12.717142 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.717117    4960 generic.go:356] "Generic (PLEG): container finished" podID="fb57c6c2-5791-478a-bfa2-c98fedc041ed" containerID="cf2b5d51ee3a9449fd8f7f34bf24de2fcc39c09d6bd5468fc8590d863c5d4673" exitCode=0
Oct 30 03:49:12.717142 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.717140    4960 generic.go:356] "Generic (PLEG): container finished" podID="fb57c6c2-5791-478a-bfa2-c98fedc041ed" containerID="4d3c374d00c1fea545c169e50f690bd7195dc60ef189082375d4841c328fe6d4" exitCode=0
Oct 30 03:49:12.717304 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.717150    4960 generic.go:356] "Generic (PLEG): container finished" podID="fb57c6c2-5791-478a-bfa2-c98fedc041ed" containerID="c53458936c2a39d594173225ff5151c76a44e407fa9e425c80dcef158bb6b9c2" exitCode=0
Oct 30 03:49:12.717304 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.717202    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fb57c6c2-5791-478a-bfa2-c98fedc041ed","Type":"ContainerDied","Data":"cf2b5d51ee3a9449fd8f7f34bf24de2fcc39c09d6bd5468fc8590d863c5d4673"}
Oct 30 03:49:12.717304 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.717261    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fb57c6c2-5791-478a-bfa2-c98fedc041ed","Type":"ContainerDied","Data":"4d3c374d00c1fea545c169e50f690bd7195dc60ef189082375d4841c328fe6d4"}
Oct 30 03:49:12.717304 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.717298    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fb57c6c2-5791-478a-bfa2-c98fedc041ed","Type":"ContainerDied","Data":"c53458936c2a39d594173225ff5151c76a44e407fa9e425c80dcef158bb6b9c2"}
Oct 30 03:49:12.720352 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.720335    4960 scope.go:122] "RemoveContainer" containerID="29e404c7f2875c58d8d2c346a092aba0c06abab63dfbd37ed1eb55d12a3c2c59"
Oct 30 03:49:12.720616 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:12.720598    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e404c7f2875c58d8d2c346a092aba0c06abab63dfbd37ed1eb55d12a3c2c59\": container with ID starting with 29e404c7f2875c58d8d2c346a092aba0c06abab63dfbd37ed1eb55d12a3c2c59 not found: ID does not exist" containerID="29e404c7f2875c58d8d2c346a092aba0c06abab63dfbd37ed1eb55d12a3c2c59"
Oct 30 03:49:12.720692 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.720621    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e404c7f2875c58d8d2c346a092aba0c06abab63dfbd37ed1eb55d12a3c2c59"} err="failed to get container status \"29e404c7f2875c58d8d2c346a092aba0c06abab63dfbd37ed1eb55d12a3c2c59\": rpc error: code = NotFound desc = could not find container \"29e404c7f2875c58d8d2c346a092aba0c06abab63dfbd37ed1eb55d12a3c2c59\": container with ID starting with 29e404c7f2875c58d8d2c346a092aba0c06abab63dfbd37ed1eb55d12a3c2c59 not found: ID does not exist"
Oct 30 03:49:12.769753 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.769714    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.769703669 podStartE2EDuration="1.769703669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:12.769163055 +0000 UTC m=+4033.418211476" watchObservedRunningTime="2026-10-30 03:49:12.769703669 +0000 UTC m=+4033.418752090"
Oct 30 03:49:12.807843 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.807815    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"]
Oct 30 03:49:12.809561 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.809539    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-mdwwk"]
Oct 30 03:49:12.888699 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.888670    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:49:12.937450 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.937417    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:12.984365 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:12.984338    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:13.043347 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.043310    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") "
Oct 30 03:49:13.043525 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.043373    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb57c6c2-5791-478a-bfa2-c98fedc041ed-kube-api-access-rprrp\" (UniqueName: \"kubernetes.io/projected/fb57c6c2-5791-478a-bfa2-c98fedc041ed-kube-api-access-rprrp\") pod \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") "
Oct 30 03:49:13.043525 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.043413    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config\") pod \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") "
Oct 30 03:49:13.043525 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.043457    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") "
Oct 30 03:49:13.043525 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.043490    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") "
Oct 30 03:49:13.043923 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.043855    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-2" pod "fb57c6c2-5791-478a-bfa2-c98fedc041ed" (UID: "fb57c6c2-5791-478a-bfa2-c98fedc041ed"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:13.044534 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.044133    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-config\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-config\") pod \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") "
Oct 30 03:49:13.044534 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.044202    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") "
Oct 30 03:49:13.044534 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.044272    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") "
Oct 30 03:49:13.044534 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.044368    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") "
Oct 30 03:49:13.044534 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.044376    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-0" pod "fb57c6c2-5791-478a-bfa2-c98fedc041ed" (UID: "fb57c6c2-5791-478a-bfa2-c98fedc041ed"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:13.044534 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.044402    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-secret-prometheus-web-tls-ca\") pod \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") "
Oct 30 03:49:13.044534 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.044448    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb57c6c2-5791-478a-bfa2-c98fedc041ed-config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb57c6c2-5791-478a-bfa2-c98fedc041ed-config-out\") pod \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") "
Oct 30 03:49:13.044534 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.044482    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-thanos-prometheus-http-client-file\") pod \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") "
Oct 30 03:49:13.045532 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.044918    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-1" pod "fb57c6c2-5791-478a-bfa2-c98fedc041ed" (UID: "fb57c6c2-5791-478a-bfa2-c98fedc041ed"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:13.046414 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.044513    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb57c6c2-5791-478a-bfa2-c98fedc041ed-tls-assets\" (UniqueName: \"kubernetes.io/projected/fb57c6c2-5791-478a-bfa2-c98fedc041ed-tls-assets\") pod \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\" (UID: \"fb57c6c2-5791-478a-bfa2-c98fedc041ed\") "
Oct 30 03:49:13.046414 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.046335    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.046414 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.046359    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.046414 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.046377    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb57c6c2-5791-478a-bfa2-c98fedc041ed-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.047825 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.047793    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "fb57c6c2-5791-478a-bfa2-c98fedc041ed" (UID: "fb57c6c2-5791-478a-bfa2-c98fedc041ed"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:13.048160 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.048134    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-config" pod "fb57c6c2-5791-478a-bfa2-c98fedc041ed" (UID: "fb57c6c2-5791-478a-bfa2-c98fedc041ed"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:13.048873 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.048846    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-secret-prometheus-web-tls-ca" pod "fb57c6c2-5791-478a-bfa2-c98fedc041ed" (UID: "fb57c6c2-5791-478a-bfa2-c98fedc041ed"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:13.048873 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.048853    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb57c6c2-5791-478a-bfa2-c98fedc041ed-kube-api-access-rprrp" pod "fb57c6c2-5791-478a-bfa2-c98fedc041ed" (UID: "fb57c6c2-5791-478a-bfa2-c98fedc041ed"). InnerVolumeSpecName "kube-api-access-rprrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:13.049104 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.049076    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-thanos-prometheus-http-client-file" pod "fb57c6c2-5791-478a-bfa2-c98fedc041ed" (UID: "fb57c6c2-5791-478a-bfa2-c98fedc041ed"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:13.049222 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.049141    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb57c6c2-5791-478a-bfa2-c98fedc041ed-config-out" pod "fb57c6c2-5791-478a-bfa2-c98fedc041ed" (UID: "fb57c6c2-5791-478a-bfa2-c98fedc041ed"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:49:13.053460 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.053403    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "fb57c6c2-5791-478a-bfa2-c98fedc041ed" (UID: "fb57c6c2-5791-478a-bfa2-c98fedc041ed"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:13.053460 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.053406    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb57c6c2-5791-478a-bfa2-c98fedc041ed-tls-assets" pod "fb57c6c2-5791-478a-bfa2-c98fedc041ed" (UID: "fb57c6c2-5791-478a-bfa2-c98fedc041ed"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:13.053764 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.053739    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87" pod "fb57c6c2-5791-478a-bfa2-c98fedc041ed" (UID: "fb57c6c2-5791-478a-bfa2-c98fedc041ed"). InnerVolumeSpecName "pvc-9689c530-7f91-4584-b28e-8809bed9e799". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:49:13.057517 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.057437    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:13.064484 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.064461    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config" pod "fb57c6c2-5791-478a-bfa2-c98fedc041ed" (UID: "fb57c6c2-5791-478a-bfa2-c98fedc041ed"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:13.147422 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.147395    4960 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.147422 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.147419    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rprrp\" (UniqueName: \"kubernetes.io/projected/fb57c6c2-5791-478a-bfa2-c98fedc041ed-kube-api-access-rprrp\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.147636 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.147430    4960 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.147636 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.147439    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.147636 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.147448    4960 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.147636 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.147473    4960 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") on node \"ip-10-0-128-241.ec2.internal\" "
Oct 30 03:49:13.147636 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.147483    4960 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-secret-prometheus-web-tls-ca\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.147636 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.147497    4960 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb57c6c2-5791-478a-bfa2-c98fedc041ed-config-out\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.147636 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.147510    4960 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb57c6c2-5791-478a-bfa2-c98fedc041ed-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.147636 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.147518    4960 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb57c6c2-5791-478a-bfa2-c98fedc041ed-tls-assets\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.158771 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.158753    4960 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-9689c530-7f91-4584-b28e-8809bed9e799" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87") on node "ip-10-0-128-241.ec2.internal"
Oct 30 03:49:13.166930 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.166908    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:13.248158 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.248094    4960 reconciler_common.go:299] "Volume detached for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"csi-6f8c8e4c1326657251c30594036033694cadf25e97408e4f9d9da436f78f3fa4\""
Oct 30 03:49:13.502258 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.502167    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb" path="/var/lib/kubelet/pods/6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb/volumes"
Oct 30 03:49:13.724894 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.724858    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fb57c6c2-5791-478a-bfa2-c98fedc041ed","Type":"ContainerDied","Data":"e0f3a425a761506b268b83ec6b7e33de61b8b4d5c775c2df0d5a07e6d595e1f2"}
Oct 30 03:49:13.724894 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.724875    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:13.725138 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.724943    4960 scope.go:122] "RemoveContainer" containerID="cf2b5d51ee3a9449fd8f7f34bf24de2fcc39c09d6bd5468fc8590d863c5d4673"
Oct 30 03:49:13.726281 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.726055    4960 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/data-science-collector-collector-0" secret="" err="secret \"data-science-collector-collector-dockercfg-wtzv9\" not found"
Oct 30 03:49:13.733539 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.733511    4960 scope.go:122] "RemoveContainer" containerID="4d3c374d00c1fea545c169e50f690bd7195dc60ef189082375d4841c328fe6d4"
Oct 30 03:49:13.741517 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.741500    4960 scope.go:122] "RemoveContainer" containerID="c53458936c2a39d594173225ff5151c76a44e407fa9e425c80dcef158bb6b9c2"
Oct 30 03:49:13.749586 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.749568    4960 scope.go:122] "RemoveContainer" containerID="baa26f5b8c7a1b5429b257fa15a9166778845dee683a19d418b172c672ca33fc"
Oct 30 03:49:13.781985 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.781960    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:49:13.797380 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.797354    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:49:13.853795 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:13.853761    4960 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found
Oct 30 03:49:13.853973 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:13.853846    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-otlp-tls-certs podName:a73780b3-6931-44ea-9a04-969fe86f9d25 nodeName:}" failed. No retries permitted until 2026-10-30 03:49:14.353823504 +0000 UTC m=+4035.002871911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "a73780b3-6931-44ea-9a04-969fe86f9d25") : secret "data-science-collector-collector-headless-tls" not found
Oct 30 03:49:13.853973 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:13.853761    4960 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-tls: secret "data-science-collector-tls" not found
Oct 30 03:49:13.854258 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:13.854240    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-tls-certs podName:a73780b3-6931-44ea-9a04-969fe86f9d25 nodeName:}" failed. No retries permitted until 2026-10-30 03:49:14.354222262 +0000 UTC m=+4035.003270661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-tls-certs") pod "data-science-collector-collector-0" (UID: "a73780b3-6931-44ea-9a04-969fe86f9d25") : secret "data-science-collector-tls" not found
Oct 30 03:49:13.854424 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:13.854408    4960 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-a0f9182a: configmap "data-science-collector-collector-a0f9182a" not found
Oct 30 03:49:13.854490 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:13.854421    4960 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-monitoring-tls: secret "data-science-collector-monitoring-tls" not found
Oct 30 03:49:13.854490 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:13.854465    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a73780b3-6931-44ea-9a04-969fe86f9d25-otc-internal podName:a73780b3-6931-44ea-9a04-969fe86f9d25 nodeName:}" failed. No retries permitted until 2026-10-30 03:49:14.354449509 +0000 UTC m=+4035.003497921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/a73780b3-6931-44ea-9a04-969fe86f9d25-otc-internal") pod "data-science-collector-collector-0" (UID: "a73780b3-6931-44ea-9a04-969fe86f9d25") : configmap "data-science-collector-collector-a0f9182a" not found
Oct 30 03:49:13.854490 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:13.854487    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-monitoring-tls-certs podName:a73780b3-6931-44ea-9a04-969fe86f9d25 nodeName:}" failed. No retries permitted until 2026-10-30 03:49:14.354476972 +0000 UTC m=+4035.003525377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-tls-certs" (UniqueName: "kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-monitoring-tls-certs") pod "data-science-collector-collector-0" (UID: "a73780b3-6931-44ea-9a04-969fe86f9d25") : secret "data-science-collector-monitoring-tls" not found
Oct 30 03:49:13.875166 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.875105    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:13.981575 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:13.981542    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:14.091094 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:14.091063    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:14.358880 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:14.358792    4960 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-a0f9182a: configmap "data-science-collector-collector-a0f9182a" not found
Oct 30 03:49:14.358880 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:14.358816    4960 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found
Oct 30 03:49:14.358880 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:14.358827    4960 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-tls: secret "data-science-collector-tls" not found
Oct 30 03:49:14.358880 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:14.358871    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-otlp-tls-certs podName:a73780b3-6931-44ea-9a04-969fe86f9d25 nodeName:}" failed. No retries permitted until 2026-10-30 03:49:15.358853005 +0000 UTC m=+4036.007901408 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "a73780b3-6931-44ea-9a04-969fe86f9d25") : secret "data-science-collector-collector-headless-tls" not found
Oct 30 03:49:14.358880 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:14.358827    4960 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-monitoring-tls: secret "data-science-collector-monitoring-tls" not found
Oct 30 03:49:14.359167 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:14.358888    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a73780b3-6931-44ea-9a04-969fe86f9d25-otc-internal podName:a73780b3-6931-44ea-9a04-969fe86f9d25 nodeName:}" failed. No retries permitted until 2026-10-30 03:49:15.358880486 +0000 UTC m=+4036.007928886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/a73780b3-6931-44ea-9a04-969fe86f9d25-otc-internal") pod "data-science-collector-collector-0" (UID: "a73780b3-6931-44ea-9a04-969fe86f9d25") : configmap "data-science-collector-collector-a0f9182a" not found
Oct 30 03:49:14.359167 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:14.358902    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-tls-certs podName:a73780b3-6931-44ea-9a04-969fe86f9d25 nodeName:}" failed. No retries permitted until 2026-10-30 03:49:15.358894109 +0000 UTC m=+4036.007942538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-tls-certs") pod "data-science-collector-collector-0" (UID: "a73780b3-6931-44ea-9a04-969fe86f9d25") : secret "data-science-collector-tls" not found
Oct 30 03:49:14.359167 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:14.358918    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-monitoring-tls-certs podName:a73780b3-6931-44ea-9a04-969fe86f9d25 nodeName:}" failed. No retries permitted until 2026-10-30 03:49:15.358909585 +0000 UTC m=+4036.007958010 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-tls-certs" (UniqueName: "kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-monitoring-tls-certs") pod "data-science-collector-collector-0" (UID: "a73780b3-6931-44ea-9a04-969fe86f9d25") : secret "data-science-collector-monitoring-tls" not found
Oct 30 03:49:14.731233 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:14.731122    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="a73780b3-6931-44ea-9a04-969fe86f9d25" containerName="otc-container" containerID="cri-o://6e8f15f363828ec855b11e0fafa67be66f53ec01dc5178595371be9768e9a785" gracePeriod=30
Oct 30 03:49:14.969442 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:14.969418    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:15.064462 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.064389    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-tls-certs\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-tls-certs\") pod \"a73780b3-6931-44ea-9a04-969fe86f9d25\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") "
Oct 30 03:49:15.064462 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.064428    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-otlp-tls-certs\") pod \"a73780b3-6931-44ea-9a04-969fe86f9d25\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") "
Oct 30 03:49:15.064462 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.064449    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a73780b3-6931-44ea-9a04-969fe86f9d25-otc-internal\" (UniqueName: \"kubernetes.io/configmap/a73780b3-6931-44ea-9a04-969fe86f9d25-otc-internal\") pod \"a73780b3-6931-44ea-9a04-969fe86f9d25\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") "
Oct 30 03:49:15.064719 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.064486    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-monitoring-tls-certs\") pod \"a73780b3-6931-44ea-9a04-969fe86f9d25\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") "
Oct 30 03:49:15.064719 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.064523    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-data-science-collector-ta-client-cert\") pod \"a73780b3-6931-44ea-9a04-969fe86f9d25\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") "
Oct 30 03:49:15.064719 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.064561    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a73780b3-6931-44ea-9a04-969fe86f9d25-kube-api-access-8pj92\" (UniqueName: \"kubernetes.io/projected/a73780b3-6931-44ea-9a04-969fe86f9d25-kube-api-access-8pj92\") pod \"a73780b3-6931-44ea-9a04-969fe86f9d25\" (UID: \"a73780b3-6931-44ea-9a04-969fe86f9d25\") "
Oct 30 03:49:15.064951 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.064923    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73780b3-6931-44ea-9a04-969fe86f9d25-otc-internal" pod "a73780b3-6931-44ea-9a04-969fe86f9d25" (UID: "a73780b3-6931-44ea-9a04-969fe86f9d25"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:15.066887 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.066854    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-monitoring-tls-certs" pod "a73780b3-6931-44ea-9a04-969fe86f9d25" (UID: "a73780b3-6931-44ea-9a04-969fe86f9d25"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:15.067016 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.066911    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73780b3-6931-44ea-9a04-969fe86f9d25-kube-api-access-8pj92" pod "a73780b3-6931-44ea-9a04-969fe86f9d25" (UID: "a73780b3-6931-44ea-9a04-969fe86f9d25"). InnerVolumeSpecName "kube-api-access-8pj92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:15.067016 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.066919    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-tls-certs" pod "a73780b3-6931-44ea-9a04-969fe86f9d25" (UID: "a73780b3-6931-44ea-9a04-969fe86f9d25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:15.067168 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.067154    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-otlp-tls-certs" pod "a73780b3-6931-44ea-9a04-969fe86f9d25" (UID: "a73780b3-6931-44ea-9a04-969fe86f9d25"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:15.067273 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.067257    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-data-science-collector-ta-client-cert" pod "a73780b3-6931-44ea-9a04-969fe86f9d25" (UID: "a73780b3-6931-44ea-9a04-969fe86f9d25"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:15.165608 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.165578    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:15.165608 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.165602    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:15.165608 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.165612    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a73780b3-6931-44ea-9a04-969fe86f9d25-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:15.165820 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.165620    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:15.165820 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.165630    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a73780b3-6931-44ea-9a04-969fe86f9d25-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:15.165820 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.165640    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pj92\" (UniqueName: \"kubernetes.io/projected/a73780b3-6931-44ea-9a04-969fe86f9d25-kube-api-access-8pj92\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:15.286661 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.286628    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="33a822a2-33ea-4a2c-93e8-383364e7593d" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.85:9093/-/ready\": dial tcp 10.132.0.85:9093: connect: connection refused"
Oct 30 03:49:15.502222 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.502169    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb57c6c2-5791-478a-bfa2-c98fedc041ed" path="/var/lib/kubelet/pods/fb57c6c2-5791-478a-bfa2-c98fedc041ed/volumes"
Oct 30 03:49:15.739742 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.739707    4960 generic.go:356] "Generic (PLEG): container finished" podID="a73780b3-6931-44ea-9a04-969fe86f9d25" containerID="6e8f15f363828ec855b11e0fafa67be66f53ec01dc5178595371be9768e9a785" exitCode=0
Oct 30 03:49:15.740151 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.739761    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:15.740151 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.739765    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a73780b3-6931-44ea-9a04-969fe86f9d25","Type":"ContainerDied","Data":"6e8f15f363828ec855b11e0fafa67be66f53ec01dc5178595371be9768e9a785"}
Oct 30 03:49:15.740151 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.739819    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a73780b3-6931-44ea-9a04-969fe86f9d25","Type":"ContainerDied","Data":"bc3d73723da20d9f42a6b9bfb870fcd7f583c8a275e52269aa79c217f0a0a956"}
Oct 30 03:49:15.740151 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.739851    4960 scope.go:122] "RemoveContainer" containerID="6e8f15f363828ec855b11e0fafa67be66f53ec01dc5178595371be9768e9a785"
Oct 30 03:49:15.748391 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.748372    4960 scope.go:122] "RemoveContainer" containerID="6e8f15f363828ec855b11e0fafa67be66f53ec01dc5178595371be9768e9a785"
Oct 30 03:49:15.748675 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:15.748654    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8f15f363828ec855b11e0fafa67be66f53ec01dc5178595371be9768e9a785\": container with ID starting with 6e8f15f363828ec855b11e0fafa67be66f53ec01dc5178595371be9768e9a785 not found: ID does not exist" containerID="6e8f15f363828ec855b11e0fafa67be66f53ec01dc5178595371be9768e9a785"
Oct 30 03:49:15.748744 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.748688    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8f15f363828ec855b11e0fafa67be66f53ec01dc5178595371be9768e9a785"} err="failed to get container status \"6e8f15f363828ec855b11e0fafa67be66f53ec01dc5178595371be9768e9a785\": rpc error: code = NotFound desc = could not find container \"6e8f15f363828ec855b11e0fafa67be66f53ec01dc5178595371be9768e9a785\": container with ID starting with 6e8f15f363828ec855b11e0fafa67be66f53ec01dc5178595371be9768e9a785 not found: ID does not exist"
Oct 30 03:49:15.793876 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.793827    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:49:15.807426 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:15.807402    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:49:17.502217 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:17.502165    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a73780b3-6931-44ea-9a04-969fe86f9d25" path="/var/lib/kubelet/pods/a73780b3-6931-44ea-9a04-969fe86f9d25/volumes"
Oct 30 03:49:19.161588 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.161557    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:19.280455 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.280424    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:19.394750 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.394713    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:19.418551 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.418487    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"]
Oct 30 03:49:19.418813 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.418800    4960 cpu_manager.go:414] "containerMap: removing container" podUID="fb57c6c2-5791-478a-bfa2-c98fedc041ed" containerName="thanos-sidecar"
Oct 30 03:49:19.418858 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.418814    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb57c6c2-5791-478a-bfa2-c98fedc041ed" containerName="thanos-sidecar"
Oct 30 03:49:19.418858 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.418840    4960 cpu_manager.go:414] "containerMap: removing container" podUID="fb57c6c2-5791-478a-bfa2-c98fedc041ed" containerName="config-reloader"
Oct 30 03:49:19.418858 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.418845    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb57c6c2-5791-478a-bfa2-c98fedc041ed" containerName="config-reloader"
Oct 30 03:49:19.418858 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.418855    4960 cpu_manager.go:414] "containerMap: removing container" podUID="6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb" containerName="korrel8r"
Oct 30 03:49:19.418858 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.418860    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb" containerName="korrel8r"
Oct 30 03:49:19.419029 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.418869    4960 cpu_manager.go:414] "containerMap: removing container" podUID="fb57c6c2-5791-478a-bfa2-c98fedc041ed" containerName="init-config-reloader"
Oct 30 03:49:19.419029 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.418875    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb57c6c2-5791-478a-bfa2-c98fedc041ed" containerName="init-config-reloader"
Oct 30 03:49:19.419029 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.418881    4960 cpu_manager.go:414] "containerMap: removing container" podUID="fb57c6c2-5791-478a-bfa2-c98fedc041ed" containerName="prometheus"
Oct 30 03:49:19.419029 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.418886    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb57c6c2-5791-478a-bfa2-c98fedc041ed" containerName="prometheus"
Oct 30 03:49:19.419029 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.418892    4960 cpu_manager.go:414] "containerMap: removing container" podUID="a73780b3-6931-44ea-9a04-969fe86f9d25" containerName="otc-container"
Oct 30 03:49:19.419029 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.418896    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a73780b3-6931-44ea-9a04-969fe86f9d25" containerName="otc-container"
Oct 30 03:49:19.419029 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.418943    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb57c6c2-5791-478a-bfa2-c98fedc041ed" containerName="config-reloader"
Oct 30 03:49:19.419029 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.418951    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a2eaaaa-10f3-4545-8865-74ecb9ff4bfb" containerName="korrel8r"
Oct 30 03:49:19.419029 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.418957    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="a73780b3-6931-44ea-9a04-969fe86f9d25" containerName="otc-container"
Oct 30 03:49:19.419029 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.418963    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb57c6c2-5791-478a-bfa2-c98fedc041ed" containerName="prometheus"
Oct 30 03:49:19.419029 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.418969    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb57c6c2-5791-478a-bfa2-c98fedc041ed" containerName="thanos-sidecar"
Oct 30 03:49:19.423288 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.423272    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:19.445002 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.444977    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"]
Oct 30 03:49:19.456951 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.456925    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 03:49:19.457107 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.457008    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 03:49:19.457107 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.457086    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-rfrsm\""
Oct 30 03:49:19.457395 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.457369    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\""
Oct 30 03:49:19.457496 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.457480    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\""
Oct 30 03:49:19.458728 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.458709    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"]
Oct 30 03:49:19.462208 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.462191    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"
Oct 30 03:49:19.483888 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.483867    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"]
Oct 30 03:49:19.499402 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.499381    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvdqc\" (UniqueName: \"kubernetes.io/projected/878b4a31-f3e3-4ce2-9591-48db278f9cd6-kube-api-access-bvdqc\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jph2q\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:19.499498 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.499415    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jph2q\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:19.499548 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.499492    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jph2q\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:19.499548 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.499531    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jph2q\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:19.499626 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.499615    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/878b4a31-f3e3-4ce2-9591-48db278f9cd6-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jph2q\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:19.509129 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.509094    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:19.544025 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.543993    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"]
Oct 30 03:49:19.547083 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.547017    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\""
Oct 30 03:49:19.547231 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.547137    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\""
Oct 30 03:49:19.547309 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.547247    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\""
Oct 30 03:49:19.547309 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.547269    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"
Oct 30 03:49:19.547309 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.547294    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-hwr7c\""
Oct 30 03:49:19.565162 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.565140    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"]
Oct 30 03:49:19.600312 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.600292    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-j894l\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"
Oct 30 03:49:19.600413 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.600334    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/45347ea8-43a1-43dc-aa34-9671d449de5b-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-j894l\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"
Oct 30 03:49:19.600589 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.600577    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvdqc\" (UniqueName: \"kubernetes.io/projected/878b4a31-f3e3-4ce2-9591-48db278f9cd6-kube-api-access-bvdqc\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jph2q\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:19.600645 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.600631    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jph2q\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:19.600723 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.600690    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jph2q\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:19.600844 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.600730    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jph2q\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:19.600844 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.600756    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-j894l\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"
Oct 30 03:49:19.600844 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.600789    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-j894l\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"
Oct 30 03:49:19.600844 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:19.600833    4960 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found
Oct 30 03:49:19.601064 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.600869    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/878b4a31-f3e3-4ce2-9591-48db278f9cd6-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jph2q\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:19.601064 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.600912    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2zbt\" (UniqueName: \"kubernetes.io/projected/45347ea8-43a1-43dc-aa34-9671d449de5b-kube-api-access-f2zbt\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-j894l\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"
Oct 30 03:49:19.601064 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:19.600935    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-prometheus-client-cert podName:878b4a31-f3e3-4ce2-9591-48db278f9cd6 nodeName:}" failed. No retries permitted until 2026-10-30 03:49:20.100915604 +0000 UTC m=+4040.749964007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-jph2q" (UID: "878b4a31-f3e3-4ce2-9591-48db278f9cd6") : secret "prometheus-operated-tls" not found
Oct 30 03:49:19.601480 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.601462    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/878b4a31-f3e3-4ce2-9591-48db278f9cd6-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jph2q\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:19.603261 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.603241    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jph2q\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:19.603332 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.603317    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jph2q\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:19.621944 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.621920    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvdqc\" (UniqueName: \"kubernetes.io/projected/878b4a31-f3e3-4ce2-9591-48db278f9cd6-kube-api-access-bvdqc\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jph2q\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:19.622133 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.622109    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:19.647621 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.647594    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\""
Oct 30 03:49:19.647711 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.647644    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-8vv9w\""
Oct 30 03:49:19.647711 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.647654    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\""
Oct 30 03:49:19.701655 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.701609    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-j894l\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"
Oct 30 03:49:19.701655 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.701637    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-j894l\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"
Oct 30 03:49:19.701806 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.701656    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2zbt\" (UniqueName: \"kubernetes.io/projected/45347ea8-43a1-43dc-aa34-9671d449de5b-kube-api-access-f2zbt\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-j894l\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"
Oct 30 03:49:19.701806 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.701678    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8aa79632-c0b4-4d84-8106-82aee30dd932-config\") pod \"korrel8r-5cbcc89b8f-srfwg\" (UID: \"8aa79632-c0b4-4d84-8106-82aee30dd932\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"
Oct 30 03:49:19.701806 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.701694    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-j894l\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"
Oct 30 03:49:19.701806 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.701719    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/45347ea8-43a1-43dc-aa34-9671d449de5b-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-j894l\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"
Oct 30 03:49:19.701806 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.701774    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8aa79632-c0b4-4d84-8106-82aee30dd932-tls\") pod \"korrel8r-5cbcc89b8f-srfwg\" (UID: \"8aa79632-c0b4-4d84-8106-82aee30dd932\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"
Oct 30 03:49:19.701806 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.701801    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgvkz\" (UniqueName: \"kubernetes.io/projected/8aa79632-c0b4-4d84-8106-82aee30dd932-kube-api-access-bgvkz\") pod \"korrel8r-5cbcc89b8f-srfwg\" (UID: \"8aa79632-c0b4-4d84-8106-82aee30dd932\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"
Oct 30 03:49:19.702196 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.702155    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-j894l\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"
Oct 30 03:49:19.702341 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.702321    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-j894l\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"
Oct 30 03:49:19.702444 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.702429    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-j894l\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"
Oct 30 03:49:19.704066 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.704048    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/45347ea8-43a1-43dc-aa34-9671d449de5b-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-j894l\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"
Oct 30 03:49:19.722055 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.722029    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2zbt\" (UniqueName: \"kubernetes.io/projected/45347ea8-43a1-43dc-aa34-9671d449de5b-kube-api-access-f2zbt\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-j894l\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"
Oct 30 03:49:19.776367 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.776345    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"
Oct 30 03:49:19.802811 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.802788    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8aa79632-c0b4-4d84-8106-82aee30dd932-config\") pod \"korrel8r-5cbcc89b8f-srfwg\" (UID: \"8aa79632-c0b4-4d84-8106-82aee30dd932\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"
Oct 30 03:49:19.802934 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.802839    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8aa79632-c0b4-4d84-8106-82aee30dd932-tls\") pod \"korrel8r-5cbcc89b8f-srfwg\" (UID: \"8aa79632-c0b4-4d84-8106-82aee30dd932\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"
Oct 30 03:49:19.802934 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.802873    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgvkz\" (UniqueName: \"kubernetes.io/projected/8aa79632-c0b4-4d84-8106-82aee30dd932-kube-api-access-bgvkz\") pod \"korrel8r-5cbcc89b8f-srfwg\" (UID: \"8aa79632-c0b4-4d84-8106-82aee30dd932\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"
Oct 30 03:49:19.803048 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:19.803001    4960 secret.go:189] Couldn't get secret redhat-ods-monitoring/korrel8r-tls: secret "korrel8r-tls" not found
Oct 30 03:49:19.803103 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:19.803060    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa79632-c0b4-4d84-8106-82aee30dd932-tls podName:8aa79632-c0b4-4d84-8106-82aee30dd932 nodeName:}" failed. No retries permitted until 2026-10-30 03:49:20.303046197 +0000 UTC m=+4040.952094596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/8aa79632-c0b4-4d84-8106-82aee30dd932-tls") pod "korrel8r-5cbcc89b8f-srfwg" (UID: "8aa79632-c0b4-4d84-8106-82aee30dd932") : secret "korrel8r-tls" not found
Oct 30 03:49:19.803577 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.803559    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8aa79632-c0b4-4d84-8106-82aee30dd932-config\") pod \"korrel8r-5cbcc89b8f-srfwg\" (UID: \"8aa79632-c0b4-4d84-8106-82aee30dd932\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"
Oct 30 03:49:19.826288 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:19.824632    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgvkz\" (UniqueName: \"kubernetes.io/projected/8aa79632-c0b4-4d84-8106-82aee30dd932-kube-api-access-bgvkz\") pod \"korrel8r-5cbcc89b8f-srfwg\" (UID: \"8aa79632-c0b4-4d84-8106-82aee30dd932\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"
Oct 30 03:49:20.034969 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:20.034892    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:20.066817 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:20.066789    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"]
Oct 30 03:49:20.068439 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:49:20.068414    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45347ea8_43a1_43dc_aa34_9671d449de5b.slice/crio-34fdacc9d98bbcb2c1745ff194cbc4bef9a700dd836d790ed958ae9c4d069ca6 WatchSource:0}: Error finding container 34fdacc9d98bbcb2c1745ff194cbc4bef9a700dd836d790ed958ae9c4d069ca6: Status 404 returned error can't find the container with id 34fdacc9d98bbcb2c1745ff194cbc4bef9a700dd836d790ed958ae9c4d069ca6
Oct 30 03:49:20.105571 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:20.105550    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jph2q\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:20.108026 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:20.107997    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jph2q\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:20.122090 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:20.122069    4960 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:49:20.222348 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:20.222324    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:20.286649 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:20.286585    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="33a822a2-33ea-4a2c-93e8-383364e7593d" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.85:9093/-/ready\": dial tcp 10.132.0.85:9093: connect: connection refused"
Oct 30 03:49:20.307551 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:20.307521    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8aa79632-c0b4-4d84-8106-82aee30dd932-tls\") pod \"korrel8r-5cbcc89b8f-srfwg\" (UID: \"8aa79632-c0b4-4d84-8106-82aee30dd932\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"
Oct 30 03:49:20.309866 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:20.309849    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8aa79632-c0b4-4d84-8106-82aee30dd932-tls\") pod \"korrel8r-5cbcc89b8f-srfwg\" (UID: \"8aa79632-c0b4-4d84-8106-82aee30dd932\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"
Oct 30 03:49:20.336037 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:20.336004    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:20.421427 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:20.421400    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:20.460271 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:20.460248    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"
Oct 30 03:49:20.631974 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:20.631936    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"]
Oct 30 03:49:20.633035 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:49:20.633008    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod878b4a31_f3e3_4ce2_9591_48db278f9cd6.slice/crio-5ec6b7bdbe4cedbc3f86b50870a3116cef29e37a8ee3edfa0e087d9497be0c71 WatchSource:0}: Error finding container 5ec6b7bdbe4cedbc3f86b50870a3116cef29e37a8ee3edfa0e087d9497be0c71: Status 404 returned error can't find the container with id 5ec6b7bdbe4cedbc3f86b50870a3116cef29e37a8ee3edfa0e087d9497be0c71
Oct 30 03:49:20.755372 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:20.755341    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"]
Oct 30 03:49:20.756601 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:49:20.756570    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa79632_c0b4_4d84_8106_82aee30dd932.slice/crio-4c14e1bdadbcb62097890d42378bf1bbcf1374f90db0c591774489069e725407 WatchSource:0}: Error finding container 4c14e1bdadbcb62097890d42378bf1bbcf1374f90db0c591774489069e725407: Status 404 returned error can't find the container with id 4c14e1bdadbcb62097890d42378bf1bbcf1374f90db0c591774489069e725407
Oct 30 03:49:20.768234 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:20.768208    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l" event={"ID":"45347ea8-43a1-43dc-aa34-9671d449de5b","Type":"ContainerStarted","Data":"ed3fc790fa8154e5ee3b6817117eb8b126bd769cbc4fc70aaae215dc776960d8"}
Oct 30 03:49:20.768330 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:20.768310    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l" event={"ID":"45347ea8-43a1-43dc-aa34-9671d449de5b","Type":"ContainerStarted","Data":"34fdacc9d98bbcb2c1745ff194cbc4bef9a700dd836d790ed958ae9c4d069ca6"}
Oct 30 03:49:20.769721 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:20.769695    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg" event={"ID":"8aa79632-c0b4-4d84-8106-82aee30dd932","Type":"ContainerStarted","Data":"4c14e1bdadbcb62097890d42378bf1bbcf1374f90db0c591774489069e725407"}
Oct 30 03:49:20.771284 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:20.771247    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q" event={"ID":"878b4a31-f3e3-4ce2-9591-48db278f9cd6","Type":"ContainerStarted","Data":"5ec6b7bdbe4cedbc3f86b50870a3116cef29e37a8ee3edfa0e087d9497be0c71"}
Oct 30 03:49:21.009963 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.009930    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:49:21.014761 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.014728    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.026653 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.026626    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:21.045555 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.045531    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:49:21.053389 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.053044    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:49:21.090315 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.090286    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:49:21.090315 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.090307    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:49:21.090315 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.090286    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:49:21.090532 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.090416    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:49:21.090908 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.090894    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-sg6sg\""
Oct 30 03:49:21.116641 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.115193    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gwcf\" (UniqueName: \"kubernetes.io/projected/5391014a-83af-4be2-b6ad-ee37b78dd484-kube-api-access-6gwcf\") pod \"data-science-collector-collector-1\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.116641 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.115289    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.116641 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.115314    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.116641 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.115405    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5391014a-83af-4be2-b6ad-ee37b78dd484-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.116641 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.115438    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.116641 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.115479    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.216235 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.216137    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5391014a-83af-4be2-b6ad-ee37b78dd484-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.216235 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.216173    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.216444 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.216245    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.216444 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.216309    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gwcf\" (UniqueName: \"kubernetes.io/projected/5391014a-83af-4be2-b6ad-ee37b78dd484-kube-api-access-6gwcf\") pod \"data-science-collector-collector-1\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.216444 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.216339    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.216444 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.216359    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.216924 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.216893    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5391014a-83af-4be2-b6ad-ee37b78dd484-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.219320 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.219298    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.219528 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.219503    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.219720 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.219702    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.220319 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.220297    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.240756 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.240710    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gwcf\" (UniqueName: \"kubernetes.io/projected/5391014a-83af-4be2-b6ad-ee37b78dd484-kube-api-access-6gwcf\") pod \"data-science-collector-collector-1\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.328421 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.328373    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:21.331342 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.331315    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:21.653489 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.653465    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:49:21.655772 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:49:21.655740    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5391014a_83af_4be2_b6ad_ee37b78dd484.slice/crio-906213609bbe43ae55b17a7f836988f85542b4af8a7328a539bfaa6ee7ba0910 WatchSource:0}: Error finding container 906213609bbe43ae55b17a7f836988f85542b4af8a7328a539bfaa6ee7ba0910: Status 404 returned error can't find the container with id 906213609bbe43ae55b17a7f836988f85542b4af8a7328a539bfaa6ee7ba0910
Oct 30 03:49:21.658155 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.658124    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:49:21.658278 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.658212    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:49:21.779037 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.779000    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l" event={"ID":"45347ea8-43a1-43dc-aa34-9671d449de5b","Type":"ContainerStarted","Data":"9b33819a834317ff7a8085afffda67f0a16d1b4f95a22275287a03c919ece6a5"}
Oct 30 03:49:21.780896 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.780857    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5391014a-83af-4be2-b6ad-ee37b78dd484","Type":"ContainerStarted","Data":"613235369d5a914988334a78e5db0e9a5243e90fa1e2d888007ac4997c775e65"}
Oct 30 03:49:21.781059 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.780918    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5391014a-83af-4be2-b6ad-ee37b78dd484","Type":"ContainerStarted","Data":"906213609bbe43ae55b17a7f836988f85542b4af8a7328a539bfaa6ee7ba0910"}
Oct 30 03:49:21.782595 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.782565    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg" event={"ID":"8aa79632-c0b4-4d84-8106-82aee30dd932","Type":"ContainerStarted","Data":"a85e3901b8c39ccd235aef3485383189fe48b5d90f427fd81b2e9c519b8ae33e"}
Oct 30 03:49:21.782794 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.782632    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"
Oct 30 03:49:21.784216 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.784192    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q" event={"ID":"878b4a31-f3e3-4ce2-9591-48db278f9cd6","Type":"ContainerStarted","Data":"ee28841f441c44af432600384f3bebdd951271f2e4e81e5e1e6fd0581ef88929"}
Oct 30 03:49:21.784327 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.784294    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:21.820846 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.820805    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l" podStartSLOduration=1.5850606539999998 podStartE2EDuration="2.820793339s" totalImagesPullingTime="1.235732685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:21.820198988 +0000 UTC m=+4042.469247410" watchObservedRunningTime="2026-10-30 03:49:21.820793339 +0000 UTC m=+4042.469841761"
Oct 30 03:49:21.864656 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.864607    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg" podStartSLOduration=2.864596186 podStartE2EDuration="2.864596186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:21.86353121 +0000 UTC m=+4042.512579631" watchObservedRunningTime="2026-10-30 03:49:21.864596186 +0000 UTC m=+4042.513644604"
Oct 30 03:49:21.904971 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.904861    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.904848718 podStartE2EDuration="1.904848718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:21.904364309 +0000 UTC m=+4042.553412723" watchObservedRunningTime="2026-10-30 03:49:21.904848718 +0000 UTC m=+4042.553897139"
Oct 30 03:49:21.945854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:21.945801    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q" podStartSLOduration=2.9457853 podStartE2EDuration="2.9457853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:21.945347705 +0000 UTC m=+4042.594396127" watchObservedRunningTime="2026-10-30 03:49:21.9457853 +0000 UTC m=+4042.594833719"
Oct 30 03:49:22.436654 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.436628    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:22.539476 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.539406    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-config-volume\" (UniqueName: \"kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-config-volume\") pod \"33a822a2-33ea-4a2c-93e8-383364e7593d\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") "
Oct 30 03:49:22.539476 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.539455    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a822a2-33ea-4a2c-93e8-383364e7593d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/33a822a2-33ea-4a2c-93e8-383364e7593d-config-out\") pod \"33a822a2-33ea-4a2c-93e8-383364e7593d\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") "
Oct 30 03:49:22.539688 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.539498    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a822a2-33ea-4a2c-93e8-383364e7593d-kube-api-access-t7xzf\" (UniqueName: \"kubernetes.io/projected/33a822a2-33ea-4a2c-93e8-383364e7593d-kube-api-access-t7xzf\") pod \"33a822a2-33ea-4a2c-93e8-383364e7593d\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") "
Oct 30 03:49:22.539688 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.539561    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-web-config\" (UniqueName: \"kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-web-config\") pod \"33a822a2-33ea-4a2c-93e8-383364e7593d\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") "
Oct 30 03:49:22.539688 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.539589    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a822a2-33ea-4a2c-93e8-383364e7593d-tls-assets\" (UniqueName: \"kubernetes.io/projected/33a822a2-33ea-4a2c-93e8-383364e7593d-tls-assets\") pod \"33a822a2-33ea-4a2c-93e8-383364e7593d\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") "
Oct 30 03:49:22.539688 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.539631    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-cluster-tls-config\") pod \"33a822a2-33ea-4a2c-93e8-383364e7593d\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") "
Oct 30 03:49:22.539688 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.539665    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a822a2-33ea-4a2c-93e8-383364e7593d-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/33a822a2-33ea-4a2c-93e8-383364e7593d-alertmanager-data-science-monitoringstack-db\") pod \"33a822a2-33ea-4a2c-93e8-383364e7593d\" (UID: \"33a822a2-33ea-4a2c-93e8-383364e7593d\") "
Oct 30 03:49:22.540308 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.540207    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a822a2-33ea-4a2c-93e8-383364e7593d-alertmanager-data-science-monitoringstack-db" pod "33a822a2-33ea-4a2c-93e8-383364e7593d" (UID: "33a822a2-33ea-4a2c-93e8-383364e7593d"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:49:22.542030 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.541995    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a822a2-33ea-4a2c-93e8-383364e7593d-config-out" pod "33a822a2-33ea-4a2c-93e8-383364e7593d" (UID: "33a822a2-33ea-4a2c-93e8-383364e7593d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:49:22.542109 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.542062    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a822a2-33ea-4a2c-93e8-383364e7593d-kube-api-access-t7xzf" pod "33a822a2-33ea-4a2c-93e8-383364e7593d" (UID: "33a822a2-33ea-4a2c-93e8-383364e7593d"). InnerVolumeSpecName "kube-api-access-t7xzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:22.542241 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.542223    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-config-volume" pod "33a822a2-33ea-4a2c-93e8-383364e7593d" (UID: "33a822a2-33ea-4a2c-93e8-383364e7593d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:22.542426 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.542395    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a822a2-33ea-4a2c-93e8-383364e7593d-tls-assets" pod "33a822a2-33ea-4a2c-93e8-383364e7593d" (UID: "33a822a2-33ea-4a2c-93e8-383364e7593d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:22.546462 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.546440    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-cluster-tls-config" pod "33a822a2-33ea-4a2c-93e8-383364e7593d" (UID: "33a822a2-33ea-4a2c-93e8-383364e7593d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:22.553982 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.553961    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-web-config" pod "33a822a2-33ea-4a2c-93e8-383364e7593d" (UID: "33a822a2-33ea-4a2c-93e8-383364e7593d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:22.640737 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.640708    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7xzf\" (UniqueName: \"kubernetes.io/projected/33a822a2-33ea-4a2c-93e8-383364e7593d-kube-api-access-t7xzf\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:22.640737 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.640734    4960 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-web-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:22.640919 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.640743    4960 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33a822a2-33ea-4a2c-93e8-383364e7593d-tls-assets\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:22.640919 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.640751    4960 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-cluster-tls-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:22.640919 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.640762    4960 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/33a822a2-33ea-4a2c-93e8-383364e7593d-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:22.640919 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.640771    4960 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/33a822a2-33ea-4a2c-93e8-383364e7593d-config-volume\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:22.640919 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.640778    4960 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33a822a2-33ea-4a2c-93e8-383364e7593d-config-out\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:49:22.791488 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.791353    4960 generic.go:356] "Generic (PLEG): container finished" podID="33a822a2-33ea-4a2c-93e8-383364e7593d" containerID="36ee9f2f0cf8f33f35092439422ee951d08c265842ffc2569646213a3a48a152" exitCode=0
Oct 30 03:49:22.791488 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.791432    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"33a822a2-33ea-4a2c-93e8-383364e7593d","Type":"ContainerDied","Data":"36ee9f2f0cf8f33f35092439422ee951d08c265842ffc2569646213a3a48a152"}
Oct 30 03:49:22.791658 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.791440    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:22.791773 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.791748    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"33a822a2-33ea-4a2c-93e8-383364e7593d","Type":"ContainerDied","Data":"327defd8dab7543809c3563cc8fef9642b803f889046af84a540ff2c82c1e520"}
Oct 30 03:49:22.791866 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.791810    4960 scope.go:122] "RemoveContainer" containerID="1ae073d5f0e640139548e908ef7b4c712486362710b8a896508596acb7116215"
Oct 30 03:49:22.802538 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.802515    4960 scope.go:122] "RemoveContainer" containerID="36ee9f2f0cf8f33f35092439422ee951d08c265842ffc2569646213a3a48a152"
Oct 30 03:49:22.810486 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.810470    4960 scope.go:122] "RemoveContainer" containerID="87db361423cd2c3402715282c32b09f56510f6d5b66aaea48da1b1a55bea93bc"
Oct 30 03:49:22.817682 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.817664    4960 scope.go:122] "RemoveContainer" containerID="1ae073d5f0e640139548e908ef7b4c712486362710b8a896508596acb7116215"
Oct 30 03:49:22.817976 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:22.817954    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ae073d5f0e640139548e908ef7b4c712486362710b8a896508596acb7116215\": container with ID starting with 1ae073d5f0e640139548e908ef7b4c712486362710b8a896508596acb7116215 not found: ID does not exist" containerID="1ae073d5f0e640139548e908ef7b4c712486362710b8a896508596acb7116215"
Oct 30 03:49:22.818071 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.817984    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ae073d5f0e640139548e908ef7b4c712486362710b8a896508596acb7116215"} err="failed to get container status \"1ae073d5f0e640139548e908ef7b4c712486362710b8a896508596acb7116215\": rpc error: code = NotFound desc = could not find container \"1ae073d5f0e640139548e908ef7b4c712486362710b8a896508596acb7116215\": container with ID starting with 1ae073d5f0e640139548e908ef7b4c712486362710b8a896508596acb7116215 not found: ID does not exist"
Oct 30 03:49:22.818071 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.818000    4960 scope.go:122] "RemoveContainer" containerID="36ee9f2f0cf8f33f35092439422ee951d08c265842ffc2569646213a3a48a152"
Oct 30 03:49:22.818395 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:22.818374    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36ee9f2f0cf8f33f35092439422ee951d08c265842ffc2569646213a3a48a152\": container with ID starting with 36ee9f2f0cf8f33f35092439422ee951d08c265842ffc2569646213a3a48a152 not found: ID does not exist" containerID="36ee9f2f0cf8f33f35092439422ee951d08c265842ffc2569646213a3a48a152"
Oct 30 03:49:22.818457 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.818400    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36ee9f2f0cf8f33f35092439422ee951d08c265842ffc2569646213a3a48a152"} err="failed to get container status \"36ee9f2f0cf8f33f35092439422ee951d08c265842ffc2569646213a3a48a152\": rpc error: code = NotFound desc = could not find container \"36ee9f2f0cf8f33f35092439422ee951d08c265842ffc2569646213a3a48a152\": container with ID starting with 36ee9f2f0cf8f33f35092439422ee951d08c265842ffc2569646213a3a48a152 not found: ID does not exist"
Oct 30 03:49:22.818457 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.818415    4960 scope.go:122] "RemoveContainer" containerID="87db361423cd2c3402715282c32b09f56510f6d5b66aaea48da1b1a55bea93bc"
Oct 30 03:49:22.818651 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:22.818636    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87db361423cd2c3402715282c32b09f56510f6d5b66aaea48da1b1a55bea93bc\": container with ID starting with 87db361423cd2c3402715282c32b09f56510f6d5b66aaea48da1b1a55bea93bc not found: ID does not exist" containerID="87db361423cd2c3402715282c32b09f56510f6d5b66aaea48da1b1a55bea93bc"
Oct 30 03:49:22.818688 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.818656    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87db361423cd2c3402715282c32b09f56510f6d5b66aaea48da1b1a55bea93bc"} err="failed to get container status \"87db361423cd2c3402715282c32b09f56510f6d5b66aaea48da1b1a55bea93bc\": rpc error: code = NotFound desc = could not find container \"87db361423cd2c3402715282c32b09f56510f6d5b66aaea48da1b1a55bea93bc\": container with ID starting with 87db361423cd2c3402715282c32b09f56510f6d5b66aaea48da1b1a55bea93bc not found: ID does not exist"
Oct 30 03:49:22.847219 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.847195    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 03:49:22.860174 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:22.860154    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 03:49:23.502421 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:23.502393    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a822a2-33ea-4a2c-93e8-383364e7593d" path="/var/lib/kubelet/pods/33a822a2-33ea-4a2c-93e8-383364e7593d/volumes"
Oct 30 03:49:23.884666 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:23.884627    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:23.955015 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:23.954981    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:49:24.114315 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.114286    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:24.223309 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.223225    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:24.494743 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.491028    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 03:49:24.494743 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.491950    4960 cpu_manager.go:414] "containerMap: removing container" podUID="33a822a2-33ea-4a2c-93e8-383364e7593d" containerName="config-reloader"
Oct 30 03:49:24.494743 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.491970    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a822a2-33ea-4a2c-93e8-383364e7593d" containerName="config-reloader"
Oct 30 03:49:24.494743 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.492004    4960 cpu_manager.go:414] "containerMap: removing container" podUID="33a822a2-33ea-4a2c-93e8-383364e7593d" containerName="init-config-reloader"
Oct 30 03:49:24.494743 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.492013    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a822a2-33ea-4a2c-93e8-383364e7593d" containerName="init-config-reloader"
Oct 30 03:49:24.494743 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.492034    4960 cpu_manager.go:414] "containerMap: removing container" podUID="33a822a2-33ea-4a2c-93e8-383364e7593d" containerName="alertmanager"
Oct 30 03:49:24.494743 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.492042    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a822a2-33ea-4a2c-93e8-383364e7593d" containerName="alertmanager"
Oct 30 03:49:24.494743 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.492266    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a822a2-33ea-4a2c-93e8-383364e7593d" containerName="config-reloader"
Oct 30 03:49:24.494743 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.492285    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a822a2-33ea-4a2c-93e8-383364e7593d" containerName="alertmanager"
Oct 30 03:49:24.498492 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.496439    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:24.498781 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.498757    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.532292 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.532263    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 03:49:24.572996 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.572968    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:49:24.576721 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.576698    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.588773 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.588752    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\""
Oct 30 03:49:24.588885 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.588781    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\""
Oct 30 03:49:24.588885 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.588834    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\""
Oct 30 03:49:24.588998 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.588986    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\""
Oct 30 03:49:24.589064 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.589050    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-qbcnw\""
Oct 30 03:49:24.602609 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.602586    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:49:24.629066 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.629025    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:24.657144 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.657121    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.657262 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.657160    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dk75\" (UniqueName: \"kubernetes.io/projected/c3685681-9bd4-4f9f-a57e-f62871a7174d-kube-api-access-2dk75\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.657262 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.657202    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.657393 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.657320    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3685681-9bd4-4f9f-a57e-f62871a7174d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.657393 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.657357    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c3685681-9bd4-4f9f-a57e-f62871a7174d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.657461 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.657420    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.657461 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.657450    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3685681-9bd4-4f9f-a57e-f62871a7174d-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.667574 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.667551    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 03:49:24.667799 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.667787    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 03:49:24.667872 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.667834    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 03:49:24.689219 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.689201    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 03:49:24.689219 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.689212    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 03:49:24.689341 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.689221    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 03:49:24.689341 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.689221    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 03:49:24.689421 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.689380    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 03:49:24.689456 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.689419    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-5m2cs\""
Oct 30 03:49:24.758474 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.758334    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.758474 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.758367    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.758474 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.758402    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxkcs\" (UniqueName: \"kubernetes.io/projected/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-kube-api-access-qxkcs\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.758474 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.758462    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dk75\" (UniqueName: \"kubernetes.io/projected/c3685681-9bd4-4f9f-a57e-f62871a7174d-kube-api-access-2dk75\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.758732 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.758498    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.758732 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.758534    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.758732 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.758570    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.758732 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.758604    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3685681-9bd4-4f9f-a57e-f62871a7174d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.758949 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.758730    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c3685681-9bd4-4f9f-a57e-f62871a7174d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.758949 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.758775    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.758949 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.758854    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.758949 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.758927    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.759157 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.758990    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.759157 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.759035    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.759157 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.759091    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.759157 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.759102    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c3685681-9bd4-4f9f-a57e-f62871a7174d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.759157 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.759151    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3685681-9bd4-4f9f-a57e-f62871a7174d-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.759478 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.759454    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.759685 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.759497    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.759685 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.759543    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.761600 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.761577    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.761782 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.761740    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3685681-9bd4-4f9f-a57e-f62871a7174d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.761966 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.761942    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3685681-9bd4-4f9f-a57e-f62871a7174d-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.762217 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.762198    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.762289 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.762273    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.779067 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.779048    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dk75\" (UniqueName: \"kubernetes.io/projected/c3685681-9bd4-4f9f-a57e-f62871a7174d-kube-api-access-2dk75\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.817296 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.817272    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:24.860188 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.860150    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.860302 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.860199    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.860302 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.860232    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxkcs\" (UniqueName: \"kubernetes.io/projected/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-kube-api-access-qxkcs\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.860302 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.860278    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.860441 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.860314    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.860441 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.860360    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.860441 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.860409    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.860441 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.860433    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.860636 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.860466    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.860636 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.860494    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.860636 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.860551    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.860636 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.860582    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.861244 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.861220    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.861355 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.861165    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.861458 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.861432    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.863364 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.863336    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.863592 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.863568    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.864236 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.864165    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.864669 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.864643    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.864806 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.864783    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.865214 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.865166    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.865309 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.865240    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.866680 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.866660    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:24.885989 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:24.885963    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxkcs\" (UniqueName: \"kubernetes.io/projected/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-kube-api-access-qxkcs\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:25.136091 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:25.136065    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 03:49:25.137525 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:49:25.137498    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3685681_9bd4_4f9f_a57e_f62871a7174d.slice/crio-0c2bb8771bcf3792f902bca2f19b8797653eda5ddea1efc3e7478c1465ce4b74 WatchSource:0}: Error finding container 0c2bb8771bcf3792f902bca2f19b8797653eda5ddea1efc3e7478c1465ce4b74: Status 404 returned error can't find the container with id 0c2bb8771bcf3792f902bca2f19b8797653eda5ddea1efc3e7478c1465ce4b74
Oct 30 03:49:25.807425 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:25.807334    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c3685681-9bd4-4f9f-a57e-f62871a7174d","Type":"ContainerStarted","Data":"be09bf9b089c87cf967bfabf05e37c421fae72650ebfd43f80b338012ab404fd"}
Oct 30 03:49:25.807425 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:25.807393    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c3685681-9bd4-4f9f-a57e-f62871a7174d","Type":"ContainerStarted","Data":"0c2bb8771bcf3792f902bca2f19b8797653eda5ddea1efc3e7478c1465ce4b74"}
Oct 30 03:49:27.794094 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:27.794060    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:49:28.823524 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:28.823497    4960 generic.go:356] "Generic (PLEG): container finished" podID="c3685681-9bd4-4f9f-a57e-f62871a7174d" containerID="be09bf9b089c87cf967bfabf05e37c421fae72650ebfd43f80b338012ab404fd" exitCode=0
Oct 30 03:49:28.823826 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:28.823565    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c3685681-9bd4-4f9f-a57e-f62871a7174d","Type":"ContainerDied","Data":"be09bf9b089c87cf967bfabf05e37c421fae72650ebfd43f80b338012ab404fd"}
Oct 30 03:49:29.831591 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:29.831556    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c3685681-9bd4-4f9f-a57e-f62871a7174d","Type":"ContainerStarted","Data":"6100470bf80aec4ee905b1f4095aeea96da35fbb0d2eab8e2b53efdb88c54696"}
Oct 30 03:49:29.832064 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:29.831630    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c3685681-9bd4-4f9f-a57e-f62871a7174d","Type":"ContainerStarted","Data":"4266c3f4b10ca6b6f55cd75f8be872d9e955ef4cdd2a70f447378c739d8913f9"}
Oct 30 03:49:29.832064 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:29.831838    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:29.905617 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:29.905568    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.520783345 podStartE2EDuration="5.905554423s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.384771078s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:29.89090137 +0000 UTC m=+4050.539949793" watchObservedRunningTime="2026-10-30 03:49:29.905554423 +0000 UTC m=+4050.554602843"
Oct 30 03:49:31.506745 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:31.506709    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:31.538423 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:31.538387    4960 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:31.607390 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:31.607361    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:31.607538 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:31.607499    4960 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:31.623166 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:31.623141    4960 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") DevicePath \"csi-2479454746a6578572f425530fc81bfe97655fb31272a4af03192f10a097ebe5\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:31.727505 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:31.727476    4960 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6b615f98501c6b5529f28e27a4cb6d663be03751269539a798523d989e4446fc/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:31.755673 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:31.755652    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:31.795914 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:31.795862    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:32.115828 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:32.115803    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:49:32.117104 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:49:32.117077    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f6c7ec_8f3c_448a_9878_eaa3ccdacd1e.slice/crio-62a327a0e29cee46f8f4bb24e96b3d9ec1993ff47f7ba8fbc2590e6320349900 WatchSource:0}: Error finding container 62a327a0e29cee46f8f4bb24e96b3d9ec1993ff47f7ba8fbc2590e6320349900: Status 404 returned error can't find the container with id 62a327a0e29cee46f8f4bb24e96b3d9ec1993ff47f7ba8fbc2590e6320349900
Oct 30 03:49:32.793567 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:32.793504    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"
Oct 30 03:49:32.848404 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:32.848366    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e","Type":"ContainerStarted","Data":"9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6"}
Oct 30 03:49:32.848583 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:32.848451    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e","Type":"ContainerStarted","Data":"62a327a0e29cee46f8f4bb24e96b3d9ec1993ff47f7ba8fbc2590e6320349900"}
Oct 30 03:49:35.837557 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:35.837527    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:49:36.630583 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:36.630550    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:36.750445 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:36.750413    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:36.866300 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:36.866272    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:36.869219 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:36.869197    4960 generic.go:356] "Generic (PLEG): container finished" podID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" containerID="9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6" exitCode=0
Oct 30 03:49:36.869326 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:36.869238    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e","Type":"ContainerDied","Data":"9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6"}
Oct 30 03:49:37.876595 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:37.876560    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e","Type":"ContainerStarted","Data":"56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03"}
Oct 30 03:49:37.876992 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:37.876628    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e","Type":"ContainerStarted","Data":"bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f"}
Oct 30 03:49:37.876992 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:37.876653    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e","Type":"ContainerStarted","Data":"e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8"}
Oct 30 03:49:37.952431 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:37.952375    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=10.567317235 podStartE2EDuration="13.952361879s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.385044644s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:37.938745151 +0000 UTC m=+4058.587793562" watchObservedRunningTime="2026-10-30 03:49:37.952361879 +0000 UTC m=+4058.601410318"
Oct 30 03:49:41.796336 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:41.796299    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:42.483992 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:42.483960    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:42.683013 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:42.682977    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:43.010370 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:43.010339    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:43.123372 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:43.123337    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:46.796649 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:46.796619    4960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:46.801729 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:46.801709    4960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:46.923093 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:46.923071    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:50.351707 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:50.351675    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:53.954313 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:49:53.954261    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:49:55.996601 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:55.996566    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:56.111150 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:56.111120    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:56.486880 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:56.486848    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:56.596107 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:56.596079    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:56.854938 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:56.854907    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:57.284033 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:49:57.283954    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:01.456389 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:01.456353    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:01.589113 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:01.589080    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:02.941467 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:02.941377    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:03.048498 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:03.048437    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:03.156818 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:03.156783    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:03.371800 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:03.371767    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:07.064388 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:07.064356    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:07.180410 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:07.180377    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:07.492076 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:07.492046    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:07.612746 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:07.612717    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:07.730469 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:07.730438    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:07.920447 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:07.920419    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:12.589007 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:12.588923    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:12.708160 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:12.708130    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:14.055200 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:14.055156    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:14.169068 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:14.169044    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:14.287439 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:14.287400    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:17.778610 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:17.778577    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:18.001746 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:18.001710    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:18.112945 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:18.112923    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:18.402124 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:18.402054    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:18.531015 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:18.530983    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:18.650026 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:18.649997    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:23.602601 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:23.602566    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:23.717624 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:23.717596    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:23.954783 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:50:23.954695    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:50:24.766586 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:24.766551    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:24.888285 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:24.888252    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:25.013938 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:25.013911    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:29.070288 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:29.070256    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:29.197310 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:29.197279    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:31.138443 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:31.138411    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:31.254989 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:31.254951    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:31.375983 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:31.375955    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:34.496496 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:34.496463    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:34.619953 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:34.619917    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:36.246884 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.246849    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"]
Oct 30 03:50:36.247304 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.247073    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q" podUID="878b4a31-f3e3-4ce2-9591-48db278f9cd6" containerName="kube-rbac-proxy" containerID="cri-o://ee28841f441c44af432600384f3bebdd951271f2e4e81e5e1e6fd0581ef88929" gracePeriod=30
Oct 30 03:50:36.264067 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.264035    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"]
Oct 30 03:50:36.264364 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.264336    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l" podUID="45347ea8-43a1-43dc-aa34-9671d449de5b" containerName="kube-rbac-proxy" containerID="cri-o://ed3fc790fa8154e5ee3b6817117eb8b126bd769cbc4fc70aaae215dc776960d8" gracePeriod=30
Oct 30 03:50:36.264437 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.264326    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l" podUID="45347ea8-43a1-43dc-aa34-9671d449de5b" containerName="prom-label-proxy" containerID="cri-o://9b33819a834317ff7a8085afffda67f0a16d1b4f95a22275287a03c919ece6a5" gracePeriod=30
Oct 30 03:50:36.305226 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.305199    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"]
Oct 30 03:50:36.305404 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.305384    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg" podUID="8aa79632-c0b4-4d84-8106-82aee30dd932" containerName="korrel8r" containerID="cri-o://a85e3901b8c39ccd235aef3485383189fe48b5d90f427fd81b2e9c519b8ae33e" gracePeriod=30
Oct 30 03:50:36.559668 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.559635    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:50:36.593327 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.592804    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-prometheus-client-cert\") pod \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") "
Oct 30 03:50:36.593327 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.592875    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/878b4a31-f3e3-4ce2-9591-48db278f9cd6-kube-api-access-bvdqc\" (UniqueName: \"kubernetes.io/projected/878b4a31-f3e3-4ce2-9591-48db278f9cd6-kube-api-access-bvdqc\") pod \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") "
Oct 30 03:50:36.593327 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.592970    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-tls\" (UniqueName: \"kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-tls\") pod \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") "
Oct 30 03:50:36.593327 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.593039    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-kube-rbac-proxy-config\") pod \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") "
Oct 30 03:50:36.593327 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.593069    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/878b4a31-f3e3-4ce2-9591-48db278f9cd6-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/878b4a31-f3e3-4ce2-9591-48db278f9cd6-prometheus-ca\") pod \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\" (UID: \"878b4a31-f3e3-4ce2-9591-48db278f9cd6\") "
Oct 30 03:50:36.596681 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.593940    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/878b4a31-f3e3-4ce2-9591-48db278f9cd6-prometheus-ca" pod "878b4a31-f3e3-4ce2-9591-48db278f9cd6" (UID: "878b4a31-f3e3-4ce2-9591-48db278f9cd6"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:36.599612 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.599523    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-kube-rbac-proxy-config" pod "878b4a31-f3e3-4ce2-9591-48db278f9cd6" (UID: "878b4a31-f3e3-4ce2-9591-48db278f9cd6"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:36.599612 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.599547    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-tls" pod "878b4a31-f3e3-4ce2-9591-48db278f9cd6" (UID: "878b4a31-f3e3-4ce2-9591-48db278f9cd6"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:36.599612 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.599549    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/878b4a31-f3e3-4ce2-9591-48db278f9cd6-kube-api-access-bvdqc" pod "878b4a31-f3e3-4ce2-9591-48db278f9cd6" (UID: "878b4a31-f3e3-4ce2-9591-48db278f9cd6"). InnerVolumeSpecName "kube-api-access-bvdqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:36.599612 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.599589    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-prometheus-client-cert" pod "878b4a31-f3e3-4ce2-9591-48db278f9cd6" (UID: "878b4a31-f3e3-4ce2-9591-48db278f9cd6"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:36.607718 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.607691    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"
Oct 30 03:50:36.616603 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.616568    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"
Oct 30 03:50:36.672418 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.672377    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:36.694504 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.694050    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45347ea8-43a1-43dc-aa34-9671d449de5b-kube-api-access-f2zbt\" (UniqueName: \"kubernetes.io/projected/45347ea8-43a1-43dc-aa34-9671d449de5b-kube-api-access-f2zbt\") pod \"45347ea8-43a1-43dc-aa34-9671d449de5b\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") "
Oct 30 03:50:36.694504 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.694108    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-client-ca\" (UniqueName: \"kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-client-ca\") pod \"45347ea8-43a1-43dc-aa34-9671d449de5b\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") "
Oct 30 03:50:36.694504 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.694143    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45347ea8-43a1-43dc-aa34-9671d449de5b-tls\" (UniqueName: \"kubernetes.io/secret/45347ea8-43a1-43dc-aa34-9671d449de5b-tls\") pod \"45347ea8-43a1-43dc-aa34-9671d449de5b\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") "
Oct 30 03:50:36.694812 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.694586    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aa79632-c0b4-4d84-8106-82aee30dd932-tls\" (UniqueName: \"kubernetes.io/secret/8aa79632-c0b4-4d84-8106-82aee30dd932-tls\") pod \"8aa79632-c0b4-4d84-8106-82aee30dd932\" (UID: \"8aa79632-c0b4-4d84-8106-82aee30dd932\") "
Oct 30 03:50:36.694812 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.694607    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-client-ca" pod "45347ea8-43a1-43dc-aa34-9671d449de5b" (UID: "45347ea8-43a1-43dc-aa34-9671d449de5b"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:36.694812 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.694635    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-config\" (UniqueName: \"kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-config\") pod \"45347ea8-43a1-43dc-aa34-9671d449de5b\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") "
Oct 30 03:50:36.694812 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.694756    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa79632-c0b4-4d84-8106-82aee30dd932-kube-api-access-bgvkz\" (UniqueName: \"kubernetes.io/projected/8aa79632-c0b4-4d84-8106-82aee30dd932-kube-api-access-bgvkz\") pod \"8aa79632-c0b4-4d84-8106-82aee30dd932\" (UID: \"8aa79632-c0b4-4d84-8106-82aee30dd932\") "
Oct 30 03:50:36.694812 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.694803    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-prometheus-tls-ca\") pod \"45347ea8-43a1-43dc-aa34-9671d449de5b\" (UID: \"45347ea8-43a1-43dc-aa34-9671d449de5b\") "
Oct 30 03:50:36.695146 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.694839    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aa79632-c0b4-4d84-8106-82aee30dd932-config\" (UniqueName: \"kubernetes.io/configmap/8aa79632-c0b4-4d84-8106-82aee30dd932-config\") pod \"8aa79632-c0b4-4d84-8106-82aee30dd932\" (UID: \"8aa79632-c0b4-4d84-8106-82aee30dd932\") "
Oct 30 03:50:36.695531 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.695309    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvdqc\" (UniqueName: \"kubernetes.io/projected/878b4a31-f3e3-4ce2-9591-48db278f9cd6-kube-api-access-bvdqc\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:36.695531 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.695326    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-config" pod "45347ea8-43a1-43dc-aa34-9671d449de5b" (UID: "45347ea8-43a1-43dc-aa34-9671d449de5b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:36.695531 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.695336    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:36.695531 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.695354    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-prometheus-tls-ca" pod "45347ea8-43a1-43dc-aa34-9671d449de5b" (UID: "45347ea8-43a1-43dc-aa34-9671d449de5b"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:36.695531 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.695371    4960 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-client-ca\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:36.695531 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.695387    4960 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-kube-rbac-proxy-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:36.695531 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.695400    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/878b4a31-f3e3-4ce2-9591-48db278f9cd6-prometheus-ca\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:36.695531 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.695413    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/878b4a31-f3e3-4ce2-9591-48db278f9cd6-prometheus-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:36.696276 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.696242    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa79632-c0b4-4d84-8106-82aee30dd932-config" pod "8aa79632-c0b4-4d84-8106-82aee30dd932" (UID: "8aa79632-c0b4-4d84-8106-82aee30dd932"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:36.698996 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.698969    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa79632-c0b4-4d84-8106-82aee30dd932-tls" pod "8aa79632-c0b4-4d84-8106-82aee30dd932" (UID: "8aa79632-c0b4-4d84-8106-82aee30dd932"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:36.699099 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.699012    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45347ea8-43a1-43dc-aa34-9671d449de5b-kube-api-access-f2zbt" pod "45347ea8-43a1-43dc-aa34-9671d449de5b" (UID: "45347ea8-43a1-43dc-aa34-9671d449de5b"). InnerVolumeSpecName "kube-api-access-f2zbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:36.701931 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.701906    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45347ea8-43a1-43dc-aa34-9671d449de5b-tls" pod "45347ea8-43a1-43dc-aa34-9671d449de5b" (UID: "45347ea8-43a1-43dc-aa34-9671d449de5b"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:36.702084 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.702007    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa79632-c0b4-4d84-8106-82aee30dd932-kube-api-access-bgvkz" pod "8aa79632-c0b4-4d84-8106-82aee30dd932" (UID: "8aa79632-c0b4-4d84-8106-82aee30dd932"). InnerVolumeSpecName "kube-api-access-bgvkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:36.732997 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.732961    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 03:50:36.733306 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.733277    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="c3685681-9bd4-4f9f-a57e-f62871a7174d" containerName="alertmanager" containerID="cri-o://4266c3f4b10ca6b6f55cd75f8be872d9e955ef4cdd2a70f447378c739d8913f9" gracePeriod=120
Oct 30 03:50:36.733429 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.733307    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="c3685681-9bd4-4f9f-a57e-f62871a7174d" containerName="config-reloader" containerID="cri-o://6100470bf80aec4ee905b1f4095aeea96da35fbb0d2eab8e2b53efdb88c54696" gracePeriod=120
Oct 30 03:50:36.753049 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.753018    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:50:36.753416 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.753389    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" containerName="thanos-sidecar" containerID="cri-o://56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03" gracePeriod=600
Oct 30 03:50:36.753642 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.753383    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" containerName="prometheus" containerID="cri-o://e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8" gracePeriod=600
Oct 30 03:50:36.753642 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.753400    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" containerName="config-reloader" containerID="cri-o://bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f" gracePeriod=600
Oct 30 03:50:36.796192 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.796153    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/45347ea8-43a1-43dc-aa34-9671d449de5b-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:36.796309 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.796214    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8aa79632-c0b4-4d84-8106-82aee30dd932-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:36.796309 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.796227    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:36.796309 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.796236    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgvkz\" (UniqueName: \"kubernetes.io/projected/8aa79632-c0b4-4d84-8106-82aee30dd932-kube-api-access-bgvkz\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:36.796309 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.796246    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/45347ea8-43a1-43dc-aa34-9671d449de5b-prometheus-tls-ca\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:36.796309 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.796256    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8aa79632-c0b4-4d84-8106-82aee30dd932-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:36.796309 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.796302    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2zbt\" (UniqueName: \"kubernetes.io/projected/45347ea8-43a1-43dc-aa34-9671d449de5b-kube-api-access-f2zbt\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:36.807609 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.807574    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:36.919200 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.919141    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" containerName="prometheus" probeResult="failure" output="Get \"https://10.132.0.99:9090/-/ready\": dial tcp 10.132.0.99:9090: connect: connection refused"
Oct 30 03:50:36.936446 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:36.936417    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:37.009812 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.009791    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:50:37.048003 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.047943    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:37.098553 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.098523    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-kube-api-access-qxkcs\" (UniqueName: \"kubernetes.io/projected/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-kube-api-access-qxkcs\") pod \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") "
Oct 30 03:50:37.098700 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.098569    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-config-out\") pod \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") "
Oct 30 03:50:37.098700 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.098614    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config\") pod \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") "
Oct 30 03:50:37.098700 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.098641    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") "
Oct 30 03:50:37.098700 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.098683    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-tls-assets\" (UniqueName: \"kubernetes.io/projected/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-tls-assets\") pod \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") "
Oct 30 03:50:37.098945 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.098711    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") "
Oct 30 03:50:37.098945 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.098760    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-config\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-config\") pod \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") "
Oct 30 03:50:37.098945 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.098789    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") "
Oct 30 03:50:37.098945 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.098848    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") "
Oct 30 03:50:37.098945 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.098879    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-secret-prometheus-web-tls-ca\") pod \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") "
Oct 30 03:50:37.099241 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.098979    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") "
Oct 30 03:50:37.099241 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.099025    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") "
Oct 30 03:50:37.099241 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.099063    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-thanos-prometheus-http-client-file\") pod \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\" (UID: \"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e\") "
Oct 30 03:50:37.099241 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.099149    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-2" pod "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" (UID: "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:37.099523 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.099500    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-1" pod "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" (UID: "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:37.099908 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.099888    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.100075 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.100058    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.100174 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.100146    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-0" pod "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" (UID: "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:37.101917 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.101889    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-config-out" pod "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" (UID: "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:50:37.102447 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.102416    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-config" pod "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" (UID: "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.102598 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.102550    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-tls-assets" pod "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" (UID: "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:37.102598 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.102574    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" (UID: "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.102797 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.102677    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-thanos-prometheus-http-client-file" pod "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" (UID: "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.103224 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.103131    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" (UID: "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.103364 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.103343    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-kube-api-access-qxkcs" pod "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" (UID: "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e"). InnerVolumeSpecName "kube-api-access-qxkcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:37.104290 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.104261    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-secret-prometheus-web-tls-ca" pod "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" (UID: "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.106502 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.106482    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2" pod "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" (UID: "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e"). InnerVolumeSpecName "pvc-72ca8347-922a-464b-8594-dfac31ff3c9d". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:50:37.116344 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.116322    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config" pod "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" (UID: "37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.161263 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.161232    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:37.164441 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.164417    4960 generic.go:356] "Generic (PLEG): container finished" podID="45347ea8-43a1-43dc-aa34-9671d449de5b" containerID="9b33819a834317ff7a8085afffda67f0a16d1b4f95a22275287a03c919ece6a5" exitCode=0
Oct 30 03:50:37.164441 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.164442    4960 generic.go:356] "Generic (PLEG): container finished" podID="45347ea8-43a1-43dc-aa34-9671d449de5b" containerID="ed3fc790fa8154e5ee3b6817117eb8b126bd769cbc4fc70aaae215dc776960d8" exitCode=0
Oct 30 03:50:37.164554 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.164464    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"
Oct 30 03:50:37.164554 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.164496    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l" event={"ID":"45347ea8-43a1-43dc-aa34-9671d449de5b","Type":"ContainerDied","Data":"9b33819a834317ff7a8085afffda67f0a16d1b4f95a22275287a03c919ece6a5"}
Oct 30 03:50:37.164632 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.164557    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l" event={"ID":"45347ea8-43a1-43dc-aa34-9671d449de5b","Type":"ContainerDied","Data":"ed3fc790fa8154e5ee3b6817117eb8b126bd769cbc4fc70aaae215dc776960d8"}
Oct 30 03:50:37.164632 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.164588    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l" event={"ID":"45347ea8-43a1-43dc-aa34-9671d449de5b","Type":"ContainerDied","Data":"34fdacc9d98bbcb2c1745ff194cbc4bef9a700dd836d790ed958ae9c4d069ca6"}
Oct 30 03:50:37.164632 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.164622    4960 scope.go:122] "RemoveContainer" containerID="9b33819a834317ff7a8085afffda67f0a16d1b4f95a22275287a03c919ece6a5"
Oct 30 03:50:37.166755 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.166727    4960 generic.go:356] "Generic (PLEG): container finished" podID="c3685681-9bd4-4f9f-a57e-f62871a7174d" containerID="6100470bf80aec4ee905b1f4095aeea96da35fbb0d2eab8e2b53efdb88c54696" exitCode=0
Oct 30 03:50:37.166840 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.166797    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c3685681-9bd4-4f9f-a57e-f62871a7174d","Type":"ContainerDied","Data":"6100470bf80aec4ee905b1f4095aeea96da35fbb0d2eab8e2b53efdb88c54696"}
Oct 30 03:50:37.168109 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.168087    4960 generic.go:356] "Generic (PLEG): container finished" podID="8aa79632-c0b4-4d84-8106-82aee30dd932" containerID="a85e3901b8c39ccd235aef3485383189fe48b5d90f427fd81b2e9c519b8ae33e" exitCode=0
Oct 30 03:50:37.168223 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.168124    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg" event={"ID":"8aa79632-c0b4-4d84-8106-82aee30dd932","Type":"ContainerDied","Data":"a85e3901b8c39ccd235aef3485383189fe48b5d90f427fd81b2e9c519b8ae33e"}
Oct 30 03:50:37.168223 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.168142    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"
Oct 30 03:50:37.168223 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.168166    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg" event={"ID":"8aa79632-c0b4-4d84-8106-82aee30dd932","Type":"ContainerDied","Data":"4c14e1bdadbcb62097890d42378bf1bbcf1374f90db0c591774489069e725407"}
Oct 30 03:50:37.169645 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.169625    4960 generic.go:356] "Generic (PLEG): container finished" podID="878b4a31-f3e3-4ce2-9591-48db278f9cd6" containerID="ee28841f441c44af432600384f3bebdd951271f2e4e81e5e1e6fd0581ef88929" exitCode=0
Oct 30 03:50:37.169791 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.169674    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"
Oct 30 03:50:37.169791 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.169713    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q" event={"ID":"878b4a31-f3e3-4ce2-9591-48db278f9cd6","Type":"ContainerDied","Data":"ee28841f441c44af432600384f3bebdd951271f2e4e81e5e1e6fd0581ef88929"}
Oct 30 03:50:37.169791 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.169753    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q" event={"ID":"878b4a31-f3e3-4ce2-9591-48db278f9cd6","Type":"ContainerDied","Data":"5ec6b7bdbe4cedbc3f86b50870a3116cef29e37a8ee3edfa0e087d9497be0c71"}
Oct 30 03:50:37.172402 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.172377    4960 generic.go:356] "Generic (PLEG): container finished" podID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" containerID="56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03" exitCode=0
Oct 30 03:50:37.172514 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.172404    4960 generic.go:356] "Generic (PLEG): container finished" podID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" containerID="bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f" exitCode=0
Oct 30 03:50:37.172514 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.172416    4960 generic.go:356] "Generic (PLEG): container finished" podID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" containerID="e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8" exitCode=0
Oct 30 03:50:37.172514 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.172442    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:50:37.172514 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.172448    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e","Type":"ContainerDied","Data":"56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03"}
Oct 30 03:50:37.172514 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.172505    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e","Type":"ContainerDied","Data":"bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f"}
Oct 30 03:50:37.172730 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.172545    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e","Type":"ContainerDied","Data":"e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8"}
Oct 30 03:50:37.172730 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.172583    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e","Type":"ContainerDied","Data":"62a327a0e29cee46f8f4bb24e96b3d9ec1993ff47f7ba8fbc2590e6320349900"}
Oct 30 03:50:37.175394 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.175380    4960 scope.go:122] "RemoveContainer" containerID="ed3fc790fa8154e5ee3b6817117eb8b126bd769cbc4fc70aaae215dc776960d8"
Oct 30 03:50:37.184878 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.184861    4960 scope.go:122] "RemoveContainer" containerID="9b33819a834317ff7a8085afffda67f0a16d1b4f95a22275287a03c919ece6a5"
Oct 30 03:50:37.185149 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:50:37.185132    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b33819a834317ff7a8085afffda67f0a16d1b4f95a22275287a03c919ece6a5\": container with ID starting with 9b33819a834317ff7a8085afffda67f0a16d1b4f95a22275287a03c919ece6a5 not found: ID does not exist" containerID="9b33819a834317ff7a8085afffda67f0a16d1b4f95a22275287a03c919ece6a5"
Oct 30 03:50:37.185320 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.185155    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b33819a834317ff7a8085afffda67f0a16d1b4f95a22275287a03c919ece6a5"} err="failed to get container status \"9b33819a834317ff7a8085afffda67f0a16d1b4f95a22275287a03c919ece6a5\": rpc error: code = NotFound desc = could not find container \"9b33819a834317ff7a8085afffda67f0a16d1b4f95a22275287a03c919ece6a5\": container with ID starting with 9b33819a834317ff7a8085afffda67f0a16d1b4f95a22275287a03c919ece6a5 not found: ID does not exist"
Oct 30 03:50:37.185320 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.185172    4960 scope.go:122] "RemoveContainer" containerID="ed3fc790fa8154e5ee3b6817117eb8b126bd769cbc4fc70aaae215dc776960d8"
Oct 30 03:50:37.185457 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:50:37.185440    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed3fc790fa8154e5ee3b6817117eb8b126bd769cbc4fc70aaae215dc776960d8\": container with ID starting with ed3fc790fa8154e5ee3b6817117eb8b126bd769cbc4fc70aaae215dc776960d8 not found: ID does not exist" containerID="ed3fc790fa8154e5ee3b6817117eb8b126bd769cbc4fc70aaae215dc776960d8"
Oct 30 03:50:37.185500 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.185463    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed3fc790fa8154e5ee3b6817117eb8b126bd769cbc4fc70aaae215dc776960d8"} err="failed to get container status \"ed3fc790fa8154e5ee3b6817117eb8b126bd769cbc4fc70aaae215dc776960d8\": rpc error: code = NotFound desc = could not find container \"ed3fc790fa8154e5ee3b6817117eb8b126bd769cbc4fc70aaae215dc776960d8\": container with ID starting with ed3fc790fa8154e5ee3b6817117eb8b126bd769cbc4fc70aaae215dc776960d8 not found: ID does not exist"
Oct 30 03:50:37.185500 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.185479    4960 scope.go:122] "RemoveContainer" containerID="9b33819a834317ff7a8085afffda67f0a16d1b4f95a22275287a03c919ece6a5"
Oct 30 03:50:37.185685 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.185665    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b33819a834317ff7a8085afffda67f0a16d1b4f95a22275287a03c919ece6a5"} err="failed to get container status \"9b33819a834317ff7a8085afffda67f0a16d1b4f95a22275287a03c919ece6a5\": rpc error: code = NotFound desc = could not find container \"9b33819a834317ff7a8085afffda67f0a16d1b4f95a22275287a03c919ece6a5\": container with ID starting with 9b33819a834317ff7a8085afffda67f0a16d1b4f95a22275287a03c919ece6a5 not found: ID does not exist"
Oct 30 03:50:37.185757 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.185687    4960 scope.go:122] "RemoveContainer" containerID="ed3fc790fa8154e5ee3b6817117eb8b126bd769cbc4fc70aaae215dc776960d8"
Oct 30 03:50:37.185901 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.185885    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed3fc790fa8154e5ee3b6817117eb8b126bd769cbc4fc70aaae215dc776960d8"} err="failed to get container status \"ed3fc790fa8154e5ee3b6817117eb8b126bd769cbc4fc70aaae215dc776960d8\": rpc error: code = NotFound desc = could not find container \"ed3fc790fa8154e5ee3b6817117eb8b126bd769cbc4fc70aaae215dc776960d8\": container with ID starting with ed3fc790fa8154e5ee3b6817117eb8b126bd769cbc4fc70aaae215dc776960d8 not found: ID does not exist"
Oct 30 03:50:37.185952 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.185902    4960 scope.go:122] "RemoveContainer" containerID="a85e3901b8c39ccd235aef3485383189fe48b5d90f427fd81b2e9c519b8ae33e"
Oct 30 03:50:37.193813 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.193791    4960 scope.go:122] "RemoveContainer" containerID="a85e3901b8c39ccd235aef3485383189fe48b5d90f427fd81b2e9c519b8ae33e"
Oct 30 03:50:37.194076 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:50:37.194059    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a85e3901b8c39ccd235aef3485383189fe48b5d90f427fd81b2e9c519b8ae33e\": container with ID starting with a85e3901b8c39ccd235aef3485383189fe48b5d90f427fd81b2e9c519b8ae33e not found: ID does not exist" containerID="a85e3901b8c39ccd235aef3485383189fe48b5d90f427fd81b2e9c519b8ae33e"
Oct 30 03:50:37.194131 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.194080    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85e3901b8c39ccd235aef3485383189fe48b5d90f427fd81b2e9c519b8ae33e"} err="failed to get container status \"a85e3901b8c39ccd235aef3485383189fe48b5d90f427fd81b2e9c519b8ae33e\": rpc error: code = NotFound desc = could not find container \"a85e3901b8c39ccd235aef3485383189fe48b5d90f427fd81b2e9c519b8ae33e\": container with ID starting with a85e3901b8c39ccd235aef3485383189fe48b5d90f427fd81b2e9c519b8ae33e not found: ID does not exist"
Oct 30 03:50:37.194131 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.194093    4960 scope.go:122] "RemoveContainer" containerID="ee28841f441c44af432600384f3bebdd951271f2e4e81e5e1e6fd0581ef88929"
Oct 30 03:50:37.201352 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.201333    4960 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.201352 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.201347    4960 scope.go:122] "RemoveContainer" containerID="ee28841f441c44af432600384f3bebdd951271f2e4e81e5e1e6fd0581ef88929"
Oct 30 03:50:37.201486 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.201356    4960 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-tls-assets\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.201486 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.201427    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.201486 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.201439    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.201486 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.201455    4960 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.201486 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.201469    4960 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-secret-prometheus-web-tls-ca\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.201683 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.201500    4960 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") on node \"ip-10-0-128-241.ec2.internal\" "
Oct 30 03:50:37.201683 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.201516    4960 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.201683 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.201528    4960 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.201683 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.201537    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxkcs\" (UniqueName: \"kubernetes.io/projected/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-kube-api-access-qxkcs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.201683 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.201546    4960 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e-config-out\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.201683 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:50:37.201657    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee28841f441c44af432600384f3bebdd951271f2e4e81e5e1e6fd0581ef88929\": container with ID starting with ee28841f441c44af432600384f3bebdd951271f2e4e81e5e1e6fd0581ef88929 not found: ID does not exist" containerID="ee28841f441c44af432600384f3bebdd951271f2e4e81e5e1e6fd0581ef88929"
Oct 30 03:50:37.201683 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.201677    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee28841f441c44af432600384f3bebdd951271f2e4e81e5e1e6fd0581ef88929"} err="failed to get container status \"ee28841f441c44af432600384f3bebdd951271f2e4e81e5e1e6fd0581ef88929\": rpc error: code = NotFound desc = could not find container \"ee28841f441c44af432600384f3bebdd951271f2e4e81e5e1e6fd0581ef88929\": container with ID starting with ee28841f441c44af432600384f3bebdd951271f2e4e81e5e1e6fd0581ef88929 not found: ID does not exist"
Oct 30 03:50:37.201974 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.201693    4960 scope.go:122] "RemoveContainer" containerID="56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03"
Oct 30 03:50:37.210133 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.210114    4960 scope.go:122] "RemoveContainer" containerID="bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f"
Oct 30 03:50:37.213606 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.213591    4960 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-72ca8347-922a-464b-8594-dfac31ff3c9d" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2") on node "ip-10-0-128-241.ec2.internal"
Oct 30 03:50:37.218241 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.218224    4960 scope.go:122] "RemoveContainer" containerID="e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8"
Oct 30 03:50:37.225662 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.225645    4960 scope.go:122] "RemoveContainer" containerID="9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6"
Oct 30 03:50:37.233360 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.233321    4960 scope.go:122] "RemoveContainer" containerID="56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03"
Oct 30 03:50:37.233591 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:50:37.233573    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03\": container with ID starting with 56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03 not found: ID does not exist" containerID="56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03"
Oct 30 03:50:37.233681 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.233599    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03"} err="failed to get container status \"56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03\": rpc error: code = NotFound desc = could not find container \"56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03\": container with ID starting with 56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03 not found: ID does not exist"
Oct 30 03:50:37.233681 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.233620    4960 scope.go:122] "RemoveContainer" containerID="bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f"
Oct 30 03:50:37.233910 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:50:37.233886    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f\": container with ID starting with bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f not found: ID does not exist" containerID="bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f"
Oct 30 03:50:37.233999 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.233918    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f"} err="failed to get container status \"bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f\": rpc error: code = NotFound desc = could not find container \"bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f\": container with ID starting with bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f not found: ID does not exist"
Oct 30 03:50:37.233999 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.233939    4960 scope.go:122] "RemoveContainer" containerID="e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8"
Oct 30 03:50:37.234365 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:50:37.234328    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8\": container with ID starting with e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8 not found: ID does not exist" containerID="e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8"
Oct 30 03:50:37.234450 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.234372    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8"} err="failed to get container status \"e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8\": rpc error: code = NotFound desc = could not find container \"e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8\": container with ID starting with e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8 not found: ID does not exist"
Oct 30 03:50:37.234450 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.234389    4960 scope.go:122] "RemoveContainer" containerID="9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6"
Oct 30 03:50:37.234699 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:50:37.234677    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6\": container with ID starting with 9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6 not found: ID does not exist" containerID="9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6"
Oct 30 03:50:37.234764 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.234707    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6"} err="failed to get container status \"9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6\": rpc error: code = NotFound desc = could not find container \"9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6\": container with ID starting with 9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6 not found: ID does not exist"
Oct 30 03:50:37.234764 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.234726    4960 scope.go:122] "RemoveContainer" containerID="56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03"
Oct 30 03:50:37.235028 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.235007    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03"} err="failed to get container status \"56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03\": rpc error: code = NotFound desc = could not find container \"56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03\": container with ID starting with 56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03 not found: ID does not exist"
Oct 30 03:50:37.235110 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.235029    4960 scope.go:122] "RemoveContainer" containerID="bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f"
Oct 30 03:50:37.235320 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.235298    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f"} err="failed to get container status \"bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f\": rpc error: code = NotFound desc = could not find container \"bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f\": container with ID starting with bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f not found: ID does not exist"
Oct 30 03:50:37.235384 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.235322    4960 scope.go:122] "RemoveContainer" containerID="e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8"
Oct 30 03:50:37.235563 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.235534    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8"} err="failed to get container status \"e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8\": rpc error: code = NotFound desc = could not find container \"e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8\": container with ID starting with e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8 not found: ID does not exist"
Oct 30 03:50:37.235625 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.235566    4960 scope.go:122] "RemoveContainer" containerID="9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6"
Oct 30 03:50:37.235794 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.235779    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6"} err="failed to get container status \"9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6\": rpc error: code = NotFound desc = could not find container \"9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6\": container with ID starting with 9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6 not found: ID does not exist"
Oct 30 03:50:37.235843 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.235794    4960 scope.go:122] "RemoveContainer" containerID="56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03"
Oct 30 03:50:37.236026 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.236007    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03"} err="failed to get container status \"56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03\": rpc error: code = NotFound desc = could not find container \"56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03\": container with ID starting with 56b2efb9d9709e79bedbad92b201b72996434b8a1e3c5d9007be49aaee91dc03 not found: ID does not exist"
Oct 30 03:50:37.236070 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.236027    4960 scope.go:122] "RemoveContainer" containerID="bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f"
Oct 30 03:50:37.236195 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.236162    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"]
Oct 30 03:50:37.236331 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.236315    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f"} err="failed to get container status \"bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f\": rpc error: code = NotFound desc = could not find container \"bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f\": container with ID starting with bd77b79407ef961eb57a3d7b711c23978d9bced18c840ea9dab9e6e0df2b4b2f not found: ID does not exist"
Oct 30 03:50:37.236375 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.236332    4960 scope.go:122] "RemoveContainer" containerID="e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8"
Oct 30 03:50:37.236727 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.236603    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8"} err="failed to get container status \"e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8\": rpc error: code = NotFound desc = could not find container \"e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8\": container with ID starting with e3901e31190dfd14eba8d8634d603bd70f3dad73269a4ad93cd2c48cf585f0d8 not found: ID does not exist"
Oct 30 03:50:37.236727 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.236630    4960 scope.go:122] "RemoveContainer" containerID="9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6"
Oct 30 03:50:37.236921 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.236900    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6"} err="failed to get container status \"9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6\": rpc error: code = NotFound desc = could not find container \"9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6\": container with ID starting with 9a468cc68a1564987799a4b21297993c106f73a29d679f439f8d679cfad0d2e6 not found: ID does not exist"
Oct 30 03:50:37.239370 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.239347    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-j894l"]
Oct 30 03:50:37.302205 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.302162    4960 reconciler_common.go:299] "Volume detached for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"csi-2479454746a6578572f425530fc81bfe97655fb31272a4af03192f10a097ebe5\""
Oct 30 03:50:37.308484 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.308465    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"]
Oct 30 03:50:37.312267 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.312249    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jph2q"]
Oct 30 03:50:37.349699 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.349165    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"]
Oct 30 03:50:37.350221 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.350203    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-srfwg"]
Oct 30 03:50:37.390152 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.390132    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:50:37.394116 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.394099    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:50:37.454215 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.451770    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:50:37.454215 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.451987    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="5391014a-83af-4be2-b6ad-ee37b78dd484" containerName="otc-container" containerID="cri-o://613235369d5a914988334a78e5db0e9a5243e90fa1e2d888007ac4997c775e65" gracePeriod=30
Oct 30 03:50:37.502208 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.502148    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:37.504041 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.504015    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" path="/var/lib/kubelet/pods/37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e/volumes"
Oct 30 03:50:37.504549 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.504533    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45347ea8-43a1-43dc-aa34-9671d449de5b" path="/var/lib/kubelet/pods/45347ea8-43a1-43dc-aa34-9671d449de5b/volumes"
Oct 30 03:50:37.505188 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.505159    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="878b4a31-f3e3-4ce2-9591-48db278f9cd6" path="/var/lib/kubelet/pods/878b4a31-f3e3-4ce2-9591-48db278f9cd6/volumes"
Oct 30 03:50:37.505537 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.505524    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aa79632-c0b4-4d84-8106-82aee30dd932" path="/var/lib/kubelet/pods/8aa79632-c0b4-4d84-8106-82aee30dd932/volumes"
Oct 30 03:50:37.619775 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.619735    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:37.715886 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.715860    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:50:37.734838 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.734815    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:37.806076 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.806044    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-data-science-collector-ta-client-cert\") pod \"5391014a-83af-4be2-b6ad-ee37b78dd484\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") "
Oct 30 03:50:37.806256 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.806094    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-monitoring-tls-certs\") pod \"5391014a-83af-4be2-b6ad-ee37b78dd484\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") "
Oct 30 03:50:37.806256 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.806140    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-tls-certs\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-tls-certs\") pod \"5391014a-83af-4be2-b6ad-ee37b78dd484\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") "
Oct 30 03:50:37.806256 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.806198    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-otlp-tls-certs\") pod \"5391014a-83af-4be2-b6ad-ee37b78dd484\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") "
Oct 30 03:50:37.806256 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.806226    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5391014a-83af-4be2-b6ad-ee37b78dd484-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5391014a-83af-4be2-b6ad-ee37b78dd484-otc-internal\") pod \"5391014a-83af-4be2-b6ad-ee37b78dd484\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") "
Oct 30 03:50:37.806474 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.806260    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5391014a-83af-4be2-b6ad-ee37b78dd484-kube-api-access-6gwcf\" (UniqueName: \"kubernetes.io/projected/5391014a-83af-4be2-b6ad-ee37b78dd484-kube-api-access-6gwcf\") pod \"5391014a-83af-4be2-b6ad-ee37b78dd484\" (UID: \"5391014a-83af-4be2-b6ad-ee37b78dd484\") "
Oct 30 03:50:37.806679 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.806642    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5391014a-83af-4be2-b6ad-ee37b78dd484-otc-internal" pod "5391014a-83af-4be2-b6ad-ee37b78dd484" (UID: "5391014a-83af-4be2-b6ad-ee37b78dd484"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:37.809084 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.809056    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-monitoring-tls-certs" pod "5391014a-83af-4be2-b6ad-ee37b78dd484" (UID: "5391014a-83af-4be2-b6ad-ee37b78dd484"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.809210 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.809079    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-tls-certs" pod "5391014a-83af-4be2-b6ad-ee37b78dd484" (UID: "5391014a-83af-4be2-b6ad-ee37b78dd484"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.809210 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.809082    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5391014a-83af-4be2-b6ad-ee37b78dd484-kube-api-access-6gwcf" pod "5391014a-83af-4be2-b6ad-ee37b78dd484" (UID: "5391014a-83af-4be2-b6ad-ee37b78dd484"). InnerVolumeSpecName "kube-api-access-6gwcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:37.809210 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.809108    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-data-science-collector-ta-client-cert" pod "5391014a-83af-4be2-b6ad-ee37b78dd484" (UID: "5391014a-83af-4be2-b6ad-ee37b78dd484"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.809210 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.809134    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-otlp-tls-certs" pod "5391014a-83af-4be2-b6ad-ee37b78dd484" (UID: "5391014a-83af-4be2-b6ad-ee37b78dd484"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.907523 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.907462    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.907523 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.907486    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.907523 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.907497    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.907523 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.907505    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5391014a-83af-4be2-b6ad-ee37b78dd484-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.907523 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.907512    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5391014a-83af-4be2-b6ad-ee37b78dd484-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.907523 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:37.907520    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gwcf\" (UniqueName: \"kubernetes.io/projected/5391014a-83af-4be2-b6ad-ee37b78dd484-kube-api-access-6gwcf\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:38.179948 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:38.179885    4960 generic.go:356] "Generic (PLEG): container finished" podID="5391014a-83af-4be2-b6ad-ee37b78dd484" containerID="613235369d5a914988334a78e5db0e9a5243e90fa1e2d888007ac4997c775e65" exitCode=0
Oct 30 03:50:38.180069 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:38.179950    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:50:38.180069 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:38.179971    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5391014a-83af-4be2-b6ad-ee37b78dd484","Type":"ContainerDied","Data":"613235369d5a914988334a78e5db0e9a5243e90fa1e2d888007ac4997c775e65"}
Oct 30 03:50:38.180069 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:38.180058    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5391014a-83af-4be2-b6ad-ee37b78dd484","Type":"ContainerDied","Data":"906213609bbe43ae55b17a7f836988f85542b4af8a7328a539bfaa6ee7ba0910"}
Oct 30 03:50:38.180233 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:38.180089    4960 scope.go:122] "RemoveContainer" containerID="613235369d5a914988334a78e5db0e9a5243e90fa1e2d888007ac4997c775e65"
Oct 30 03:50:38.189173 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:38.189148    4960 scope.go:122] "RemoveContainer" containerID="613235369d5a914988334a78e5db0e9a5243e90fa1e2d888007ac4997c775e65"
Oct 30 03:50:38.189474 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:50:38.189455    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613235369d5a914988334a78e5db0e9a5243e90fa1e2d888007ac4997c775e65\": container with ID starting with 613235369d5a914988334a78e5db0e9a5243e90fa1e2d888007ac4997c775e65 not found: ID does not exist" containerID="613235369d5a914988334a78e5db0e9a5243e90fa1e2d888007ac4997c775e65"
Oct 30 03:50:38.189528 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:38.189481    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613235369d5a914988334a78e5db0e9a5243e90fa1e2d888007ac4997c775e65"} err="failed to get container status \"613235369d5a914988334a78e5db0e9a5243e90fa1e2d888007ac4997c775e65\": rpc error: code = NotFound desc = could not find container \"613235369d5a914988334a78e5db0e9a5243e90fa1e2d888007ac4997c775e65\": container with ID starting with 613235369d5a914988334a78e5db0e9a5243e90fa1e2d888007ac4997c775e65 not found: ID does not exist"
Oct 30 03:50:38.232963 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:38.232943    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:50:38.236423 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:38.236403    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:50:38.418951 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:38.418923    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:38.523549 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:38.523470    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:38.638366 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:38.638332    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:39.502222 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:39.502167    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5391014a-83af-4be2-b6ad-ee37b78dd484" path="/var/lib/kubelet/pods/5391014a-83af-4be2-b6ad-ee37b78dd484/volumes"
Oct 30 03:50:39.962119 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:39.962081    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:40.837082 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:40.837046    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="c3685681-9bd4-4f9f-a57e-f62871a7174d" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.98:9093/-/ready\": dial tcp 10.132.0.98:9093: connect: connection refused"
Oct 30 03:50:45.837154 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:45.837109    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="c3685681-9bd4-4f9f-a57e-f62871a7174d" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.98:9093/-/ready\": dial tcp 10.132.0.98:9093: connect: connection refused"
Oct 30 03:50:46.980864 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:46.980843    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:50:47.074919 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.074896    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-config-volume\" (UniqueName: \"kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-config-volume\") pod \"c3685681-9bd4-4f9f-a57e-f62871a7174d\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") "
Oct 30 03:50:47.075065 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.074961    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3685681-9bd4-4f9f-a57e-f62871a7174d-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c3685681-9bd4-4f9f-a57e-f62871a7174d-alertmanager-data-science-monitoringstack-db\") pod \"c3685681-9bd4-4f9f-a57e-f62871a7174d\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") "
Oct 30 03:50:47.075065 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.074993    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3685681-9bd4-4f9f-a57e-f62871a7174d-tls-assets\" (UniqueName: \"kubernetes.io/projected/c3685681-9bd4-4f9f-a57e-f62871a7174d-tls-assets\") pod \"c3685681-9bd4-4f9f-a57e-f62871a7174d\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") "
Oct 30 03:50:47.075065 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.075014    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-web-config\" (UniqueName: \"kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-web-config\") pod \"c3685681-9bd4-4f9f-a57e-f62871a7174d\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") "
Oct 30 03:50:47.075065 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.075041    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3685681-9bd4-4f9f-a57e-f62871a7174d-kube-api-access-2dk75\" (UniqueName: \"kubernetes.io/projected/c3685681-9bd4-4f9f-a57e-f62871a7174d-kube-api-access-2dk75\") pod \"c3685681-9bd4-4f9f-a57e-f62871a7174d\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") "
Oct 30 03:50:47.075318 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.075077    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3685681-9bd4-4f9f-a57e-f62871a7174d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3685681-9bd4-4f9f-a57e-f62871a7174d-config-out\") pod \"c3685681-9bd4-4f9f-a57e-f62871a7174d\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") "
Oct 30 03:50:47.075318 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.075158    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-cluster-tls-config\") pod \"c3685681-9bd4-4f9f-a57e-f62871a7174d\" (UID: \"c3685681-9bd4-4f9f-a57e-f62871a7174d\") "
Oct 30 03:50:47.075638 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.075610    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3685681-9bd4-4f9f-a57e-f62871a7174d-alertmanager-data-science-monitoringstack-db" pod "c3685681-9bd4-4f9f-a57e-f62871a7174d" (UID: "c3685681-9bd4-4f9f-a57e-f62871a7174d"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:50:47.077539 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.077511    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3685681-9bd4-4f9f-a57e-f62871a7174d-kube-api-access-2dk75" pod "c3685681-9bd4-4f9f-a57e-f62871a7174d" (UID: "c3685681-9bd4-4f9f-a57e-f62871a7174d"). InnerVolumeSpecName "kube-api-access-2dk75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:47.077768 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.077744    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3685681-9bd4-4f9f-a57e-f62871a7174d-config-out" pod "c3685681-9bd4-4f9f-a57e-f62871a7174d" (UID: "c3685681-9bd4-4f9f-a57e-f62871a7174d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:50:47.077917 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.077900    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3685681-9bd4-4f9f-a57e-f62871a7174d-tls-assets" pod "c3685681-9bd4-4f9f-a57e-f62871a7174d" (UID: "c3685681-9bd4-4f9f-a57e-f62871a7174d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:47.078108 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.078088    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-config-volume" pod "c3685681-9bd4-4f9f-a57e-f62871a7174d" (UID: "c3685681-9bd4-4f9f-a57e-f62871a7174d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:47.082008 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.081978    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-cluster-tls-config" pod "c3685681-9bd4-4f9f-a57e-f62871a7174d" (UID: "c3685681-9bd4-4f9f-a57e-f62871a7174d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:47.089033 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.089007    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-web-config" pod "c3685681-9bd4-4f9f-a57e-f62871a7174d" (UID: "c3685681-9bd4-4f9f-a57e-f62871a7174d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:47.176457 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.176433    4960 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c3685681-9bd4-4f9f-a57e-f62871a7174d-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:47.176457 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.176455    4960 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3685681-9bd4-4f9f-a57e-f62871a7174d-tls-assets\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:47.176576 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.176465    4960 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-web-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:47.176576 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.176474    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dk75\" (UniqueName: \"kubernetes.io/projected/c3685681-9bd4-4f9f-a57e-f62871a7174d-kube-api-access-2dk75\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:47.176576 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.176483    4960 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3685681-9bd4-4f9f-a57e-f62871a7174d-config-out\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:47.176576 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.176491    4960 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-cluster-tls-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:47.176576 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.176499    4960 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3685681-9bd4-4f9f-a57e-f62871a7174d-config-volume\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 03:50:47.229283 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.229259    4960 generic.go:356] "Generic (PLEG): container finished" podID="c3685681-9bd4-4f9f-a57e-f62871a7174d" containerID="4266c3f4b10ca6b6f55cd75f8be872d9e955ef4cdd2a70f447378c739d8913f9" exitCode=0
Oct 30 03:50:47.229380 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.229291    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c3685681-9bd4-4f9f-a57e-f62871a7174d","Type":"ContainerDied","Data":"4266c3f4b10ca6b6f55cd75f8be872d9e955ef4cdd2a70f447378c739d8913f9"}
Oct 30 03:50:47.229380 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.229317    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:50:47.229380 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.229333    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c3685681-9bd4-4f9f-a57e-f62871a7174d","Type":"ContainerDied","Data":"0c2bb8771bcf3792f902bca2f19b8797653eda5ddea1efc3e7478c1465ce4b74"}
Oct 30 03:50:47.229380 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.229358    4960 scope.go:122] "RemoveContainer" containerID="6100470bf80aec4ee905b1f4095aeea96da35fbb0d2eab8e2b53efdb88c54696"
Oct 30 03:50:47.237718 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.237694    4960 scope.go:122] "RemoveContainer" containerID="4266c3f4b10ca6b6f55cd75f8be872d9e955ef4cdd2a70f447378c739d8913f9"
Oct 30 03:50:47.245313 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.245291    4960 scope.go:122] "RemoveContainer" containerID="be09bf9b089c87cf967bfabf05e37c421fae72650ebfd43f80b338012ab404fd"
Oct 30 03:50:47.252909 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.252892    4960 scope.go:122] "RemoveContainer" containerID="6100470bf80aec4ee905b1f4095aeea96da35fbb0d2eab8e2b53efdb88c54696"
Oct 30 03:50:47.253202 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:50:47.253164    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6100470bf80aec4ee905b1f4095aeea96da35fbb0d2eab8e2b53efdb88c54696\": container with ID starting with 6100470bf80aec4ee905b1f4095aeea96da35fbb0d2eab8e2b53efdb88c54696 not found: ID does not exist" containerID="6100470bf80aec4ee905b1f4095aeea96da35fbb0d2eab8e2b53efdb88c54696"
Oct 30 03:50:47.253289 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.253210    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6100470bf80aec4ee905b1f4095aeea96da35fbb0d2eab8e2b53efdb88c54696"} err="failed to get container status \"6100470bf80aec4ee905b1f4095aeea96da35fbb0d2eab8e2b53efdb88c54696\": rpc error: code = NotFound desc = could not find container \"6100470bf80aec4ee905b1f4095aeea96da35fbb0d2eab8e2b53efdb88c54696\": container with ID starting with 6100470bf80aec4ee905b1f4095aeea96da35fbb0d2eab8e2b53efdb88c54696 not found: ID does not exist"
Oct 30 03:50:47.253289 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.253233    4960 scope.go:122] "RemoveContainer" containerID="4266c3f4b10ca6b6f55cd75f8be872d9e955ef4cdd2a70f447378c739d8913f9"
Oct 30 03:50:47.253520 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:50:47.253500    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4266c3f4b10ca6b6f55cd75f8be872d9e955ef4cdd2a70f447378c739d8913f9\": container with ID starting with 4266c3f4b10ca6b6f55cd75f8be872d9e955ef4cdd2a70f447378c739d8913f9 not found: ID does not exist" containerID="4266c3f4b10ca6b6f55cd75f8be872d9e955ef4cdd2a70f447378c739d8913f9"
Oct 30 03:50:47.253558 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.253527    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4266c3f4b10ca6b6f55cd75f8be872d9e955ef4cdd2a70f447378c739d8913f9"} err="failed to get container status \"4266c3f4b10ca6b6f55cd75f8be872d9e955ef4cdd2a70f447378c739d8913f9\": rpc error: code = NotFound desc = could not find container \"4266c3f4b10ca6b6f55cd75f8be872d9e955ef4cdd2a70f447378c739d8913f9\": container with ID starting with 4266c3f4b10ca6b6f55cd75f8be872d9e955ef4cdd2a70f447378c739d8913f9 not found: ID does not exist"
Oct 30 03:50:47.253558 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.253544    4960 scope.go:122] "RemoveContainer" containerID="be09bf9b089c87cf967bfabf05e37c421fae72650ebfd43f80b338012ab404fd"
Oct 30 03:50:47.253782 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:50:47.253765    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be09bf9b089c87cf967bfabf05e37c421fae72650ebfd43f80b338012ab404fd\": container with ID starting with be09bf9b089c87cf967bfabf05e37c421fae72650ebfd43f80b338012ab404fd not found: ID does not exist" containerID="be09bf9b089c87cf967bfabf05e37c421fae72650ebfd43f80b338012ab404fd"
Oct 30 03:50:47.253836 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.253791    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be09bf9b089c87cf967bfabf05e37c421fae72650ebfd43f80b338012ab404fd"} err="failed to get container status \"be09bf9b089c87cf967bfabf05e37c421fae72650ebfd43f80b338012ab404fd\": rpc error: code = NotFound desc = could not find container \"be09bf9b089c87cf967bfabf05e37c421fae72650ebfd43f80b338012ab404fd\": container with ID starting with be09bf9b089c87cf967bfabf05e37c421fae72650ebfd43f80b338012ab404fd not found: ID does not exist"
Oct 30 03:50:47.285012 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.284994    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 03:50:47.288363 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.288345    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 03:50:47.503027 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:47.502970    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3685681-9bd4-4f9f-a57e-f62871a7174d" path="/var/lib/kubelet/pods/c3685681-9bd4-4f9f-a57e-f62871a7174d/volumes"
Oct 30 03:50:53.953568 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:50:53.953532    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:50:57.402094 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:57.402067    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:59.667153 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:59.667115    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:59.793552 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:59.793508    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:59.934517 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:50:59.934440    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:00.031274 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031244    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"]
Oct 30 03:51:00.031556 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031544    4960 cpu_manager.go:414] "containerMap: removing container" podUID="5391014a-83af-4be2-b6ad-ee37b78dd484" containerName="otc-container"
Oct 30 03:51:00.031611 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031557    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5391014a-83af-4be2-b6ad-ee37b78dd484" containerName="otc-container"
Oct 30 03:51:00.031611 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031565    4960 cpu_manager.go:414] "containerMap: removing container" podUID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" containerName="init-config-reloader"
Oct 30 03:51:00.031611 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031571    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" containerName="init-config-reloader"
Oct 30 03:51:00.031611 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031587    4960 cpu_manager.go:414] "containerMap: removing container" podUID="878b4a31-f3e3-4ce2-9591-48db278f9cd6" containerName="kube-rbac-proxy"
Oct 30 03:51:00.031611 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031593    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="878b4a31-f3e3-4ce2-9591-48db278f9cd6" containerName="kube-rbac-proxy"
Oct 30 03:51:00.031611 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031602    4960 cpu_manager.go:414] "containerMap: removing container" podUID="45347ea8-43a1-43dc-aa34-9671d449de5b" containerName="prom-label-proxy"
Oct 30 03:51:00.031611 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031606    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45347ea8-43a1-43dc-aa34-9671d449de5b" containerName="prom-label-proxy"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031619    4960 cpu_manager.go:414] "containerMap: removing container" podUID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" containerName="prometheus"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031624    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" containerName="prometheus"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031632    4960 cpu_manager.go:414] "containerMap: removing container" podUID="45347ea8-43a1-43dc-aa34-9671d449de5b" containerName="kube-rbac-proxy"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031637    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45347ea8-43a1-43dc-aa34-9671d449de5b" containerName="kube-rbac-proxy"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031644    4960 cpu_manager.go:414] "containerMap: removing container" podUID="8aa79632-c0b4-4d84-8106-82aee30dd932" containerName="korrel8r"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031648    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa79632-c0b4-4d84-8106-82aee30dd932" containerName="korrel8r"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031655    4960 cpu_manager.go:414] "containerMap: removing container" podUID="c3685681-9bd4-4f9f-a57e-f62871a7174d" containerName="alertmanager"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031661    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3685681-9bd4-4f9f-a57e-f62871a7174d" containerName="alertmanager"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031669    4960 cpu_manager.go:414] "containerMap: removing container" podUID="c3685681-9bd4-4f9f-a57e-f62871a7174d" containerName="init-config-reloader"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031674    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3685681-9bd4-4f9f-a57e-f62871a7174d" containerName="init-config-reloader"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031681    4960 cpu_manager.go:414] "containerMap: removing container" podUID="c3685681-9bd4-4f9f-a57e-f62871a7174d" containerName="config-reloader"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031685    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3685681-9bd4-4f9f-a57e-f62871a7174d" containerName="config-reloader"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031692    4960 cpu_manager.go:414] "containerMap: removing container" podUID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" containerName="config-reloader"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031696    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" containerName="config-reloader"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031702    4960 cpu_manager.go:414] "containerMap: removing container" podUID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" containerName="thanos-sidecar"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031706    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" containerName="thanos-sidecar"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031754    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" containerName="prometheus"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031763    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="45347ea8-43a1-43dc-aa34-9671d449de5b" containerName="kube-rbac-proxy"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031769    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa79632-c0b4-4d84-8106-82aee30dd932" containerName="korrel8r"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031776    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3685681-9bd4-4f9f-a57e-f62871a7174d" containerName="alertmanager"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031781    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3685681-9bd4-4f9f-a57e-f62871a7174d" containerName="config-reloader"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031786    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" containerName="config-reloader"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031791    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f6c7ec-8f3c-448a-9878-eaa3ccdacd1e" containerName="thanos-sidecar"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031797    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="5391014a-83af-4be2-b6ad-ee37b78dd484" containerName="otc-container"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031804    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="878b4a31-f3e3-4ce2-9591-48db278f9cd6" containerName="kube-rbac-proxy"
Oct 30 03:51:00.031854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.031808    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="45347ea8-43a1-43dc-aa34-9671d449de5b" containerName="prom-label-proxy"
Oct 30 03:51:00.036478 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.036460    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 03:51:00.054825 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.054800    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"]
Oct 30 03:51:00.057397 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.057343    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:00.066835 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.066795    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:51:00.066835 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.066795    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:51:00.066835 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.066831    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ca-bundle\""
Oct 30 03:51:00.069346 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.069329    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-config\""
Oct 30 03:51:00.100548 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.100528    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-query-frontend-http\""
Oct 30 03:51:00.100663 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.100551    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-dockercfg-ssglj\""
Oct 30 03:51:00.100663 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.100557    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-query-frontend-grpc\""
Oct 30 03:51:00.174468 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.174440    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0fc7b99-6a35-40a6-840b-8d8e578474be-config\") pod \"data-science-lokistack-query-frontend-f746fc66d-m8gc4\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 03:51:00.174573 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.174484    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-f746fc66d-m8gc4\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 03:51:00.174573 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.174505    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-f746fc66d-m8gc4\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 03:51:00.174690 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.174658    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-f746fc66d-m8gc4\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 03:51:00.174773 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.174754    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9fs5\" (UniqueName: \"kubernetes.io/projected/a0fc7b99-6a35-40a6-840b-8d8e578474be-kube-api-access-l9fs5\") pod \"data-science-lokistack-query-frontend-f746fc66d-m8gc4\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 03:51:00.184455 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.184433    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:00.275852 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.275787    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-f746fc66d-m8gc4\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 03:51:00.275852 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.275819    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-f746fc66d-m8gc4\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 03:51:00.276022 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.275863    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-f746fc66d-m8gc4\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 03:51:00.276022 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.275915    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9fs5\" (UniqueName: \"kubernetes.io/projected/a0fc7b99-6a35-40a6-840b-8d8e578474be-kube-api-access-l9fs5\") pod \"data-science-lokistack-query-frontend-f746fc66d-m8gc4\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 03:51:00.276022 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.275950    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0fc7b99-6a35-40a6-840b-8d8e578474be-config\") pod \"data-science-lokistack-query-frontend-f746fc66d-m8gc4\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 03:51:00.276638 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.276608    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-f746fc66d-m8gc4\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 03:51:00.277039 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.277021    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0fc7b99-6a35-40a6-840b-8d8e578474be-config\") pod \"data-science-lokistack-query-frontend-f746fc66d-m8gc4\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 03:51:00.278556 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.278533    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-f746fc66d-m8gc4\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 03:51:00.278726 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.278706    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-f746fc66d-m8gc4\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 03:51:00.297876 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.297850    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9fs5\" (UniqueName: \"kubernetes.io/projected/a0fc7b99-6a35-40a6-840b-8d8e578474be-kube-api-access-l9fs5\") pod \"data-science-lokistack-query-frontend-f746fc66d-m8gc4\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 03:51:00.308524 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.308496    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"]
Oct 30 03:51:00.311287 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.311148    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:00.314774 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.314756    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.332519 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.332491    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"]
Oct 30 03:51:00.343146 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.343128    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\""
Oct 30 03:51:00.343302 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.343281    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-ca-bundle\""
Oct 30 03:51:00.350381 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.350363    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 03:51:00.376577 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.376551    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.376689 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.376611    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/48a4dd86-7b31-4634-9531-30286c829729-tenants\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.376763 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.376685    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm2z2\" (UniqueName: \"kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-kube-api-access-tm2z2\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.376763 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.376726    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-tls-secret\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.376854 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.376822    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-rbac\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.376894 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.376876    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-lokistack-gateway\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.376948 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.376913    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.377055 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.377037    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.380492 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.380452    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-dockercfg-ndbrm\""
Oct 30 03:51:00.380492 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.380452    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\""
Oct 30 03:51:00.380650 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.380545    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-client-http\""
Oct 30 03:51:00.380650 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.380598    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-http\""
Oct 30 03:51:00.422780 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.422692    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:00.477674 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.477647    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-lokistack-gateway\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.477821 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.477697    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.477821 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.477738    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.477821 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.477773    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.477821 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.477803    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/48a4dd86-7b31-4634-9531-30286c829729-tenants\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.478026 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.477908    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm2z2\" (UniqueName: \"kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-kube-api-access-tm2z2\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.478026 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.477952    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-tls-secret\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.478026 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.477991    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-rbac\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.478567 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.478517    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.478567 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.478552    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-lokistack-gateway\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.478718 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.478674    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-rbac\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.480336 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.480314    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.480783 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.480761    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-tls-secret\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.480965 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.480950    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/48a4dd86-7b31-4634-9531-30286c829729-tenants\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.480999 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.480955    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.501489 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.501471    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm2z2\" (UniqueName: \"kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-kube-api-access-tm2z2\") pod \"data-science-lokistack-gateway-7c4fb6d468-hmmfj\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.539876 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.539801    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:00.630037 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.630009    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:00.642070 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.642047    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"]
Oct 30 03:51:00.642925 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:51:00.642900    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0fc7b99_6a35_40a6_840b_8d8e578474be.slice/crio-9b5b7c8750747ff67f8fcc264b94976c1bd58717db418f9b3740fead9ea5f047 WatchSource:0}: Error finding container 9b5b7c8750747ff67f8fcc264b94976c1bd58717db418f9b3740fead9ea5f047: Status 404 returned error can't find the container with id 9b5b7c8750747ff67f8fcc264b94976c1bd58717db418f9b3740fead9ea5f047
Oct 30 03:51:00.932991 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:00.932965    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"]
Oct 30 03:51:00.934996 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:51:00.934973    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a4dd86_7b31_4634_9531_30286c829729.slice/crio-123f69b1437003e3cdf12395df325444c85115a84dd1761bd0c101b9fc0f17cf WatchSource:0}: Error finding container 123f69b1437003e3cdf12395df325444c85115a84dd1761bd0c101b9fc0f17cf: Status 404 returned error can't find the container with id 123f69b1437003e3cdf12395df325444c85115a84dd1761bd0c101b9fc0f17cf
Oct 30 03:51:01.300641 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:01.300579    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4" event={"ID":"a0fc7b99-6a35-40a6-840b-8d8e578474be","Type":"ContainerStarted","Data":"9b5b7c8750747ff67f8fcc264b94976c1bd58717db418f9b3740fead9ea5f047"}
Oct 30 03:51:01.301926 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:01.301901    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj" event={"ID":"48a4dd86-7b31-4634-9531-30286c829729","Type":"ContainerStarted","Data":"123f69b1437003e3cdf12395df325444c85115a84dd1761bd0c101b9fc0f17cf"}
Oct 30 03:51:02.129413 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:02.129364    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:02.231583 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:02.231543    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:02.829393 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:02.829358    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"]
Oct 30 03:51:02.834516 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:02.834487    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:02.852809 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:02.852782    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-shared-secret\""
Oct 30 03:51:02.865923 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:02.865899    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-http\""
Oct 30 03:51:02.866049 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:02.866001    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-grpc\""
Oct 30 03:51:02.867403 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:02.867383    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"]
Oct 30 03:51:03.002545 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.002511    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:03.002726 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.002556    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwhwl\" (UniqueName: \"kubernetes.io/projected/647414b0-c92a-445a-8451-8d7272daac7a-kube-api-access-cwhwl\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:03.002726 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.002605    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/647414b0-c92a-445a-8451-8d7272daac7a-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:03.002726 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.002653    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-test-loki-shared-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:03.002726 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.002696    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:03.002726 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.002724    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:03.014821 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.014751    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:03.103648 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.103577    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:03.103648 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.103624    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:03.103878 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.103691    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:03.103878 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.103720    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhwl\" (UniqueName: \"kubernetes.io/projected/647414b0-c92a-445a-8451-8d7272daac7a-kube-api-access-cwhwl\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:03.103878 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.103773    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/647414b0-c92a-445a-8451-8d7272daac7a-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:03.103878 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.103824    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-test-loki-shared-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:03.104749 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.104720    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:03.105299 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.105271    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/647414b0-c92a-445a-8451-8d7272daac7a-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:03.106770 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.106740    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:03.106948 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.106924    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-test-loki-shared-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:03.107902 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.107875    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:03.125972 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.125948    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwhwl\" (UniqueName: \"kubernetes.io/projected/647414b0-c92a-445a-8451-8d7272daac7a-kube-api-access-cwhwl\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:03.147308 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.147275    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:03.302167 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.302117    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:03.416342 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.416251    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:03.994997 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:03.994955    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"]
Oct 30 03:51:04.000096 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.000067    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:04.018795 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.018766    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"]
Oct 30 03:51:04.030746 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.030696    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-compactor-http\""
Oct 30 03:51:04.030856 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.030791    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-compactor-grpc\""
Oct 30 03:51:04.114686 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.114653    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a91f7d11-3482-4103-a72b-570a1c8f5019-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:04.114892 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.114741    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:04.114892 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.114776    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-test-loki-shared-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:04.115015 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.114907    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:04.115015 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.114968    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:04.115117 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.115030    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxszr\" (UniqueName: \"kubernetes.io/projected/a91f7d11-3482-4103-a72b-570a1c8f5019-kube-api-access-zxszr\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:04.120382 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.120349    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:04.216364 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.216339    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:04.216796 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.216371    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxszr\" (UniqueName: \"kubernetes.io/projected/a91f7d11-3482-4103-a72b-570a1c8f5019-kube-api-access-zxszr\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:04.216796 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.216533    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a91f7d11-3482-4103-a72b-570a1c8f5019-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:04.216796 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.216583    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:04.216796 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.216624    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-test-loki-shared-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:04.216796 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.216683    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:04.217404 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.217355    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:04.217679 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.217418    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a91f7d11-3482-4103-a72b-570a1c8f5019-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:04.219199 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.219161    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:04.219599 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.219575    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:04.219685 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.219643    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-test-loki-shared-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:04.238153 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.238126    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxszr\" (UniqueName: \"kubernetes.io/projected/a91f7d11-3482-4103-a72b-570a1c8f5019-kube-api-access-zxszr\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:04.305717 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.305677    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:04.329097 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.328749    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4" event={"ID":"a0fc7b99-6a35-40a6-840b-8d8e578474be","Type":"ContainerStarted","Data":"e16a7f2d4e64b5adf9ce424b563fe9827c8440ffa74f6382b82b6a1ebba2ebec"}
Oct 30 03:51:04.329097 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.328811    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 03:51:04.330928 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.330899    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj" event={"ID":"48a4dd86-7b31-4634-9531-30286c829729","Type":"ContainerStarted","Data":"eb8b7362f46fab949aa799a4b8ad1be1c7989e3a910a69e0447bb866ba88d675"}
Oct 30 03:51:04.370570 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:04.370521    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4" podStartSLOduration=0.870044024 podStartE2EDuration="4.370507478s" totalImagesPullingTime="3.500463454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:04.369702629 +0000 UTC m=+4145.018751087" watchObservedRunningTime="2026-10-30 03:51:04.370507478 +0000 UTC m=+4145.019555899"
Oct 30 03:51:06.342932 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:06.342893    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj" event={"ID":"48a4dd86-7b31-4634-9531-30286c829729","Type":"ContainerStarted","Data":"7cd8a5d433616bc4c7cefcda80e86ac94e870fa1e6c624bd1a02ddb8b22ed46c"}
Oct 30 03:51:06.343356 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:06.343088    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:06.349101 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:06.349078    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:06.402754 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:06.402699    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj" podStartSLOduration=1.552161721 podStartE2EDuration="6.402687082s" totalImagesPullingTime="4.850525361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:06.401639775 +0000 UTC m=+4147.050688195" watchObservedRunningTime="2026-10-30 03:51:06.402687082 +0000 UTC m=+4147.051735502"
Oct 30 03:51:06.637715 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:06.637652    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:06.637715 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:06.637686    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4575788d-4b47-476e-89a1-a05676f07b7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:06.668626 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:06.668596    4960 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4575788d-4b47-476e-89a1-a05676f07b7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:06.668724 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:06.668647    4960 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:06.739062 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:06.739030    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:06.739062 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:06.739064    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4575788d-4b47-476e-89a1-a05676f07b7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:06.739288 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:06.739161    4960 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:06.739288 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:06.739164    4960 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4575788d-4b47-476e-89a1-a05676f07b7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:06.754394 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:06.754369    4960 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") DevicePath \"csi-79b8ce0216485321c872c3dd9c6696ead485e08c1d139e467da9818fbf093e5a\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:06.754394 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:06.754391    4960 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4575788d-4b47-476e-89a1-a05676f07b7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") DevicePath \"csi-07e2e97f2246d642a8e47279f26d69dde46412f7218628a5c9d916389d78b478\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:06.988675 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:06.988644    4960 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/24cb2292f4f1a6dc2e1530c8f0fcd532bb4fab1a28a419ddab25747e5cf108bd/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:07.014316 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:07.014291    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:07.348011 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:07.347983    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:07.356041 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:07.356021    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 03:51:13.894937 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:13.894905    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-35cf2c66-f433-429c-932a-bb5f878f69a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:13.926366 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:13.926335    4960 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-35cf2c66-f433-429c-932a-bb5f878f69a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:13.995468 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:13.995441    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-35cf2c66-f433-429c-932a-bb5f878f69a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:13.995603 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:13.995553    4960 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-35cf2c66-f433-429c-932a-bb5f878f69a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:14.010846 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:14.010820    4960 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-35cf2c66-f433-429c-932a-bb5f878f69a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") DevicePath \"csi-f5257c3efa9a6c12782ddbc06dda04660bd20ea308d42e7920bd1a2ecc0f0594\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:14.074098 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:14.074076    4960 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4575788d-4b47-476e-89a1-a05676f07b7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/eb1c0e88ab61acf9c34bfdc8d9f5526591662938fd3d7e5232b08417061ca4a2/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:14.099502 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:14.099481    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4575788d-4b47-476e-89a1-a05676f07b7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") pod \"data-science-lokistack-ingester-0\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:14.239502 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:14.239477    4960 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-35cf2c66-f433-429c-932a-bb5f878f69a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a18a593227275cf9c0182e316398b7a2c249d11f860a234713a451ed47265e62/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:14.249564 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:14.249545    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:14.265846 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:14.265822    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-35cf2c66-f433-429c-932a-bb5f878f69a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") pod \"data-science-lokistack-compactor-0\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:14.515733 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:14.515663    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:14.552852 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:14.552823    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"]
Oct 30 03:51:14.554053 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:51:14.554023    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647414b0_c92a_445a_8451_8d7272daac7a.slice/crio-04c2aa0e8cbc4f07e7afca8d7f3511534a032f455949ba59bfc33fa88e201498 WatchSource:0}: Error finding container 04c2aa0e8cbc4f07e7afca8d7f3511534a032f455949ba59bfc33fa88e201498: Status 404 returned error can't find the container with id 04c2aa0e8cbc4f07e7afca8d7f3511534a032f455949ba59bfc33fa88e201498
Oct 30 03:51:14.810427 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:14.810400    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"]
Oct 30 03:51:14.811818 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:51:14.811794    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda91f7d11_3482_4103_a72b_570a1c8f5019.slice/crio-37cb346fcb4c2204ee1b98bc258322b8df97f8c15ea5b5ae9db12ba03910e5f2 WatchSource:0}: Error finding container 37cb346fcb4c2204ee1b98bc258322b8df97f8c15ea5b5ae9db12ba03910e5f2: Status 404 returned error can't find the container with id 37cb346fcb4c2204ee1b98bc258322b8df97f8c15ea5b5ae9db12ba03910e5f2
Oct 30 03:51:15.387461 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:15.387425    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"a91f7d11-3482-4103-a72b-570a1c8f5019","Type":"ContainerStarted","Data":"6a80c9e15f94365dd5efc73ce0f5514b82f98a86ab0fa4a8005830e4395c6041"}
Oct 30 03:51:15.387874 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:15.387473    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"a91f7d11-3482-4103-a72b-570a1c8f5019","Type":"ContainerStarted","Data":"37cb346fcb4c2204ee1b98bc258322b8df97f8c15ea5b5ae9db12ba03910e5f2"}
Oct 30 03:51:15.387874 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:15.387539    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:15.389076 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:15.389056    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"647414b0-c92a-445a-8451-8d7272daac7a","Type":"ContainerStarted","Data":"bd42531b69165c218b6fd1587d1e8692a12a324a3051c084f88ea160bdef5da5"}
Oct 30 03:51:15.389221 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:15.389087    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"647414b0-c92a-445a-8451-8d7272daac7a","Type":"ContainerStarted","Data":"04c2aa0e8cbc4f07e7afca8d7f3511534a032f455949ba59bfc33fa88e201498"}
Oct 30 03:51:15.389221 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:15.389105    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:51:15.432077 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:15.432034    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" podStartSLOduration=16.432023728 podStartE2EDuration="16.432023728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:15.430700546 +0000 UTC m=+4156.079748964" watchObservedRunningTime="2026-10-30 03:51:15.432023728 +0000 UTC m=+4156.081072148"
Oct 30 03:51:15.475610 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:15.475555    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podStartSLOduration=16.475547036 podStartE2EDuration="16.475547036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:15.474551278 +0000 UTC m=+4156.123599712" watchObservedRunningTime="2026-10-30 03:51:15.475547036 +0000 UTC m=+4156.124595457"
Oct 30 03:51:23.954569 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:51:23.954521    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:51:25.343119 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:25.343084    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 03:51:32.046997 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:32.046958    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:32.152727 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:32.152698    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:34.339174 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:34.339141    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:34.490964 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:34.490932    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:35.161219 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:35.161164    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:36.399754 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:36.399720    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="647414b0-c92a-445a-8451-8d7272daac7a" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:51:36.400193 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:36.400161    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 03:51:46.403678 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:46.403643    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="647414b0-c92a-445a-8451-8d7272daac7a" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:51:53.953391 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:51:53.953349    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:51:56.399208 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:56.399142    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="647414b0-c92a-445a-8451-8d7272daac7a" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:51:57.511745 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:57.511709    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:59.911038 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:59.911012    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:51:59.922161 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:59.922137    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:51:59.922477 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:59.922145    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:51:59.928784 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:51:59.928765    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:52:06.398303 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:52:06.398271    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 03:52:06.548425 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:52:06.548379    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:52:06.657849 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:52:06.657774    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:52:08.827726 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:52:08.827692    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:52:08.935438 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:52:08.935409    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:52:09.667528 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:52:09.667470    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:52:23.952851 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:52:23.952802    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:52:53.953882 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:52:53.953842    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:52:57.648916 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:52:57.648887    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:18.381857 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:53:18.381757    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:18.491718 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:53:18.491687    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:20.668097 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:53:20.668052    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:20.797049 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:53:20.797015    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:21.495074 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:53:21.495045    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:21.599706 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:53:21.599677    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:23.956815 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:53:23.956775    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:53:42.245832 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:53:42.245800    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:53.954754 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:53:53.954709    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:53:56.831367 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:53:56.831331    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:56.935801 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:53:56.935773    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:57.745791 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:53:57.745742    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:23.401091 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:23.401049    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:23.956897 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:54:23.956861    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:54:39.178499 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.178462    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz"]
Oct 30 03:54:39.183457 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.183434    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz"
Oct 30 03:54:39.203382 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.203355    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-60155212\""
Oct 30 03:54:39.204738 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.204716    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz"]
Oct 30 03:54:39.212340 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.212308    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv8vp\" (UniqueName: \"kubernetes.io/projected/7c720a2d-5aef-4479-b7f2-ce543d96bfea-kube-api-access-lv8vp\") pod \"data-science-usage-logs-collector-7b866bc69f-cz8bz\" (UID: \"7c720a2d-5aef-4479-b7f2-ce543d96bfea\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz"
Oct 30 03:54:39.212452 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.212374    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7c720a2d-5aef-4479-b7f2-ce543d96bfea-otc-internal\") pod \"data-science-usage-logs-collector-7b866bc69f-cz8bz\" (UID: \"7c720a2d-5aef-4479-b7f2-ce543d96bfea\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz"
Oct 30 03:54:39.212524 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.212491    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c720a2d-5aef-4479-b7f2-ce543d96bfea-otlp-tls-certs\") pod \"data-science-usage-logs-collector-7b866bc69f-cz8bz\" (UID: \"7c720a2d-5aef-4479-b7f2-ce543d96bfea\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz"
Oct 30 03:54:39.216967 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.216945    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-headless-tls\""
Oct 30 03:54:39.217161 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.217144    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-4t4c9\""
Oct 30 03:54:39.313564 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.313533    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7c720a2d-5aef-4479-b7f2-ce543d96bfea-otc-internal\") pod \"data-science-usage-logs-collector-7b866bc69f-cz8bz\" (UID: \"7c720a2d-5aef-4479-b7f2-ce543d96bfea\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz"
Oct 30 03:54:39.313724 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.313589    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c720a2d-5aef-4479-b7f2-ce543d96bfea-otlp-tls-certs\") pod \"data-science-usage-logs-collector-7b866bc69f-cz8bz\" (UID: \"7c720a2d-5aef-4479-b7f2-ce543d96bfea\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz"
Oct 30 03:54:39.313724 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.313654    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv8vp\" (UniqueName: \"kubernetes.io/projected/7c720a2d-5aef-4479-b7f2-ce543d96bfea-kube-api-access-lv8vp\") pod \"data-science-usage-logs-collector-7b866bc69f-cz8bz\" (UID: \"7c720a2d-5aef-4479-b7f2-ce543d96bfea\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz"
Oct 30 03:54:39.314267 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.314240    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7c720a2d-5aef-4479-b7f2-ce543d96bfea-otc-internal\") pod \"data-science-usage-logs-collector-7b866bc69f-cz8bz\" (UID: \"7c720a2d-5aef-4479-b7f2-ce543d96bfea\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz"
Oct 30 03:54:39.316169 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.316146    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c720a2d-5aef-4479-b7f2-ce543d96bfea-otlp-tls-certs\") pod \"data-science-usage-logs-collector-7b866bc69f-cz8bz\" (UID: \"7c720a2d-5aef-4479-b7f2-ce543d96bfea\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz"
Oct 30 03:54:39.335533 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.335511    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:39.338461 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.338441    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv8vp\" (UniqueName: \"kubernetes.io/projected/7c720a2d-5aef-4479-b7f2-ce543d96bfea-kube-api-access-lv8vp\") pod \"data-science-usage-logs-collector-7b866bc69f-cz8bz\" (UID: \"7c720a2d-5aef-4479-b7f2-ce543d96bfea\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz"
Oct 30 03:54:39.457550 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.457495    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:39.497642 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.497619    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz"
Oct 30 03:54:39.791436 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.791411    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz"]
Oct 30 03:54:39.793272 ip-10-0-128-241 kubenswrapper[4960]: W1030 03:54:39.793241    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c720a2d_5aef_4479_b7f2_ce543d96bfea.slice/crio-8ede93cf89e0a222f9ee117b1137de0068c6b05efb1917be9f3f5710786e2fdc WatchSource:0}: Error finding container 8ede93cf89e0a222f9ee117b1137de0068c6b05efb1917be9f3f5710786e2fdc: Status 404 returned error can't find the container with id 8ede93cf89e0a222f9ee117b1137de0068c6b05efb1917be9f3f5710786e2fdc
Oct 30 03:54:39.795433 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.795386    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:54:39.795501 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:39.795475    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 03:54:40.308750 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:40.308713    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz" event={"ID":"7c720a2d-5aef-4479-b7f2-ce543d96bfea","Type":"ContainerStarted","Data":"582178cea61625bd34a5314fb0dc16e4ed6d98ca8623e6cdb63abdfeb1917cc2"}
Oct 30 03:54:40.309268 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:40.308777    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz" event={"ID":"7c720a2d-5aef-4479-b7f2-ce543d96bfea","Type":"ContainerStarted","Data":"8ede93cf89e0a222f9ee117b1137de0068c6b05efb1917be9f3f5710786e2fdc"}
Oct 30 03:54:40.351081 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:40.351036    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz" podStartSLOduration=1.351021948 podStartE2EDuration="1.351021948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:54:40.349486647 +0000 UTC m=+4360.998535068" watchObservedRunningTime="2026-10-30 03:54:40.351021948 +0000 UTC m=+4361.000070368"
Oct 30 03:54:53.953344 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:54:53.953306    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:54:57.862239 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:54:57.862210    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:02.206572 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:55:02.206538    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:02.321038 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:55:02.321008    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:23.953733 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:55:23.953698    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:55:48.351802 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:55:48.351763    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:48.463450 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:55:48.463426    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:53.952982 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:55:53.952944    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:55:57.969823 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:55:57.969789    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:23.956823 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:56:23.956709    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:56:53.953835 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:56:53.953799    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:56:58.137388 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:56:58.137351    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:59.948088 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:56:59.948062    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:56:59.953850 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:56:59.953828    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:56:59.954604 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:56:59.954579    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 03:56:59.959840 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:56:59.959823    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 03:57:11.478617 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:57:11.478582    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:23.953384 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:57:23.953344    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:57:24.597116 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:57:24.597082    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:53.953130 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:57:53.953093    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:57:58.225192 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:57:58.225157    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:23.953317 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:58:23.953283    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:58:53.953977 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:58:53.953937    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:58:58.338080 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:58:58.338044    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 03:59:23.953609 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:59:23.953533    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:59:53.952729 ip-10-0-128-241 kubenswrapper[4960]: E1030 03:59:53.952693    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 03:59:58.409754 ip-10-0-128-241 kubenswrapper[4960]: I1030 03:59:58.409724    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:23.953384 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:00:23.953346    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:00:53.952716 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:00:53.952678    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:00:58.551889 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:00:58.551841    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:01:23.952728 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:01:23.952693    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:01:40.346918 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:01:40.346866    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:01:53.954208 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:01:53.954149    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:01:58.686365 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:01:58.686331    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:01:59.983741 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:01:59.983712    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 04:01:59.989280 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:01:59.989250    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 04:01:59.990040 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:01:59.990009    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 04:01:59.996065 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:01:59.996044    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 04:02:23.954039 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:02:23.953960    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:02:53.954155 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:02:53.954119    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:02:58.803966 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:02:58.803932    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:23.954134 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:03:23.954088    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:03:53.953893 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:03:53.953856    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:04:00.045861 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:04:00.045828    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:23.956416 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:04:23.956375    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:04:53.953746 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:04:53.953707    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:04:58.994847 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:04:58.994816    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:23.955853 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:05:23.955813    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:05:53.953297 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:05:53.953253    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:05:59.142094 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:05:59.142056    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:23.953563 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:06:23.953529    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:06:32.440799 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:06:32.440762    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:53.953899 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:06:53.953863    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:06:59.279067 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:06:59.279030    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:00.012664 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:07:00.012624    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 04:07:00.018944 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:07:00.018922    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 04:07:00.020541 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:07:00.020520    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 04:07:00.026452 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:07:00.026435    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 04:07:23.957373 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:07:23.957335    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:07:53.954271 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:07:53.954239    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:07:59.392844 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:07:59.392813    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:08:23.954124 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:08:23.954089    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:08:53.953859 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:08:53.953823    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:08:59.500461 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:08:59.500427    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:23.953451 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:23.953412    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:09:33.966001 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:33.965961    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:49.735097 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.735063    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"]
Oct 30 04:09:49.735633 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.735263    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" podUID="a91f7d11-3482-4103-a72b-570a1c8f5019" containerName="loki-compactor" containerID="cri-o://6a80c9e15f94365dd5efc73ce0f5514b82f98a86ab0fa4a8005830e4395c6041" gracePeriod=30
Oct 30 04:09:49.777737 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.777708    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"]
Oct 30 04:09:49.781310 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.781289    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:49.801736 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.801394    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-lifecycle-secret\""
Oct 30 04:09:49.802928 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.802908    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-http\""
Oct 30 04:09:49.803047 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.802961    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-grpc\""
Oct 30 04:09:49.803875 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.803852    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"]
Oct 30 04:09:49.813443 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.813407    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94a8bb9f-2c6e-42e6-b692-760ef1aee647-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-dj7xs\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:49.813443 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.813434    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-dj7xs\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:49.813593 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.813461    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-dj7xs\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:49.813593 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.813515    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-dj7xs\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:49.813680 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.813608    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-dj7xs\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:49.813680 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.813626    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7bnd\" (UniqueName: \"kubernetes.io/projected/94a8bb9f-2c6e-42e6-b692-760ef1aee647-kube-api-access-z7bnd\") pod \"data-science-lokistack-querier-56ffbd5fdc-dj7xs\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:49.914417 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.914389    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94a8bb9f-2c6e-42e6-b692-760ef1aee647-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-dj7xs\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:49.914598 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.914429    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-dj7xs\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:49.914598 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.914461    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-dj7xs\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:49.914598 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.914504    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-dj7xs\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:49.914598 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.914555    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-dj7xs\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:49.914598 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.914583    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7bnd\" (UniqueName: \"kubernetes.io/projected/94a8bb9f-2c6e-42e6-b692-760ef1aee647-kube-api-access-z7bnd\") pod \"data-science-lokistack-querier-56ffbd5fdc-dj7xs\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:49.915965 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.915940    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94a8bb9f-2c6e-42e6-b692-760ef1aee647-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-dj7xs\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:49.916133 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.916006    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-dj7xs\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:49.918022 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.917995    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-dj7xs\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:49.922676 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.922647    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-dj7xs\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:49.924403 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.924382    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-dj7xs\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:49.936217 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:49.936172    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7bnd\" (UniqueName: \"kubernetes.io/projected/94a8bb9f-2c6e-42e6-b692-760ef1aee647-kube-api-access-z7bnd\") pod \"data-science-lokistack-querier-56ffbd5fdc-dj7xs\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:50.093868 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.093835    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:50.126958 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.126930    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:50.225403 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.225376    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:50.251419 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.251352    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:50.322975 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.322945    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a91f7d11-3482-4103-a72b-570a1c8f5019-config\" (UniqueName: \"kubernetes.io/configmap/a91f7d11-3482-4103-a72b-570a1c8f5019-config\") pod \"a91f7d11-3482-4103-a72b-570a1c8f5019\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") "
Oct 30 04:09:50.323132 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.323024    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-ca-bundle\") pod \"a91f7d11-3482-4103-a72b-570a1c8f5019\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") "
Oct 30 04:09:50.323132 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.323115    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a91f7d11-3482-4103-a72b-570a1c8f5019-kube-api-access-zxszr\" (UniqueName: \"kubernetes.io/projected/a91f7d11-3482-4103-a72b-570a1c8f5019-kube-api-access-zxszr\") pod \"a91f7d11-3482-4103-a72b-570a1c8f5019\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") "
Oct 30 04:09:50.323284 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.323161    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-test-loki-shared-secret\") pod \"a91f7d11-3482-4103-a72b-570a1c8f5019\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") "
Oct 30 04:09:50.323284 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.323217    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-compactor-grpc\") pod \"a91f7d11-3482-4103-a72b-570a1c8f5019\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") "
Oct 30 04:09:50.323393 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.323325    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") pod \"a91f7d11-3482-4103-a72b-570a1c8f5019\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") "
Oct 30 04:09:50.323393 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.323386    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-compactor-http\") pod \"a91f7d11-3482-4103-a72b-570a1c8f5019\" (UID: \"a91f7d11-3482-4103-a72b-570a1c8f5019\") "
Oct 30 04:09:50.323503 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.323438    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a91f7d11-3482-4103-a72b-570a1c8f5019-config" pod "a91f7d11-3482-4103-a72b-570a1c8f5019" (UID: "a91f7d11-3482-4103-a72b-570a1c8f5019"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:50.323503 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.323445    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-ca-bundle" pod "a91f7d11-3482-4103-a72b-570a1c8f5019" (UID: "a91f7d11-3482-4103-a72b-570a1c8f5019"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:50.323747 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.323703    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a91f7d11-3482-4103-a72b-570a1c8f5019-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:50.323747 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.323731    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:50.325867 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.325842    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-test-loki-shared-secret" pod "a91f7d11-3482-4103-a72b-570a1c8f5019" (UID: "a91f7d11-3482-4103-a72b-570a1c8f5019"). InnerVolumeSpecName "test-loki-shared-secret". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:50.326021 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.326000    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91f7d11-3482-4103-a72b-570a1c8f5019-kube-api-access-zxszr" pod "a91f7d11-3482-4103-a72b-570a1c8f5019" (UID: "a91f7d11-3482-4103-a72b-570a1c8f5019"). InnerVolumeSpecName "kube-api-access-zxszr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:50.326088 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.326078    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-compactor-grpc" pod "a91f7d11-3482-4103-a72b-570a1c8f5019" (UID: "a91f7d11-3482-4103-a72b-570a1c8f5019"). InnerVolumeSpecName "data-science-lokistack-compactor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:50.326338 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.326315    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-compactor-http" pod "a91f7d11-3482-4103-a72b-570a1c8f5019" (UID: "a91f7d11-3482-4103-a72b-570a1c8f5019"). InnerVolumeSpecName "data-science-lokistack-compactor-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:50.326518 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.326500    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f" pod "a91f7d11-3482-4103-a72b-570a1c8f5019" (UID: "a91f7d11-3482-4103-a72b-570a1c8f5019"). InnerVolumeSpecName "pvc-35cf2c66-f433-429c-932a-bb5f878f69a1". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:09:50.402106 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.402085    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"]
Oct 30 04:09:50.404060 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:09:50.404039    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a8bb9f_2c6e_42e6_b692_760ef1aee647.slice/crio-49fd4cec9fc8f9f7b9507b2062dbb57366715d9fe1a8c334ccaaf844668cc78b WatchSource:0}: Error finding container 49fd4cec9fc8f9f7b9507b2062dbb57366715d9fe1a8c334ccaaf844668cc78b: Status 404 returned error can't find the container with id 49fd4cec9fc8f9f7b9507b2062dbb57366715d9fe1a8c334ccaaf844668cc78b
Oct 30 04:09:50.424294 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.424271    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxszr\" (UniqueName: \"kubernetes.io/projected/a91f7d11-3482-4103-a72b-570a1c8f5019-kube-api-access-zxszr\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:50.424294 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.424294    4960 reconciler_common.go:299] "Volume detached for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-test-loki-shared-secret\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:50.424456 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.424306    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-compactor-grpc\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:50.424456 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.424332    4960 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-35cf2c66-f433-429c-932a-bb5f878f69a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") on node \"ip-10-0-128-241.ec2.internal\" "
Oct 30 04:09:50.424456 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.424343    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/a91f7d11-3482-4103-a72b-570a1c8f5019-data-science-lokistack-compactor-http\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:50.433876 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.433861    4960 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-35cf2c66-f433-429c-932a-bb5f878f69a1" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f") on node "ip-10-0-128-241.ec2.internal"
Oct 30 04:09:50.484376 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.484351    4960 generic.go:356] "Generic (PLEG): container finished" podID="a91f7d11-3482-4103-a72b-570a1c8f5019" containerID="6a80c9e15f94365dd5efc73ce0f5514b82f98a86ab0fa4a8005830e4395c6041" exitCode=0
Oct 30 04:09:50.484495 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.484398    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:50.484495 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.484433    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"a91f7d11-3482-4103-a72b-570a1c8f5019","Type":"ContainerDied","Data":"6a80c9e15f94365dd5efc73ce0f5514b82f98a86ab0fa4a8005830e4395c6041"}
Oct 30 04:09:50.484495 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.484485    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"a91f7d11-3482-4103-a72b-570a1c8f5019","Type":"ContainerDied","Data":"37cb346fcb4c2204ee1b98bc258322b8df97f8c15ea5b5ae9db12ba03910e5f2"}
Oct 30 04:09:50.485051 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.484511    4960 scope.go:122] "RemoveContainer" containerID="6a80c9e15f94365dd5efc73ce0f5514b82f98a86ab0fa4a8005830e4395c6041"
Oct 30 04:09:50.486947 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.486921    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs" event={"ID":"94a8bb9f-2c6e-42e6-b692-760ef1aee647","Type":"ContainerStarted","Data":"55aed86839419e4d2b96e1865572ee21ecf861b2f964218d285c35021af92e18"}
Oct 30 04:09:50.487109 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.486972    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs" event={"ID":"94a8bb9f-2c6e-42e6-b692-760ef1aee647","Type":"ContainerStarted","Data":"49fd4cec9fc8f9f7b9507b2062dbb57366715d9fe1a8c334ccaaf844668cc78b"}
Oct 30 04:09:50.487288 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.487114    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:50.497986 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.497962    4960 scope.go:122] "RemoveContainer" containerID="6a80c9e15f94365dd5efc73ce0f5514b82f98a86ab0fa4a8005830e4395c6041"
Oct 30 04:09:50.498512 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:50.498429    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a80c9e15f94365dd5efc73ce0f5514b82f98a86ab0fa4a8005830e4395c6041\": container with ID starting with 6a80c9e15f94365dd5efc73ce0f5514b82f98a86ab0fa4a8005830e4395c6041 not found: ID does not exist" containerID="6a80c9e15f94365dd5efc73ce0f5514b82f98a86ab0fa4a8005830e4395c6041"
Oct 30 04:09:50.498512 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.498473    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a80c9e15f94365dd5efc73ce0f5514b82f98a86ab0fa4a8005830e4395c6041"} err="failed to get container status \"6a80c9e15f94365dd5efc73ce0f5514b82f98a86ab0fa4a8005830e4395c6041\": rpc error: code = NotFound desc = could not find container \"6a80c9e15f94365dd5efc73ce0f5514b82f98a86ab0fa4a8005830e4395c6041\": container with ID starting with 6a80c9e15f94365dd5efc73ce0f5514b82f98a86ab0fa4a8005830e4395c6041 not found: ID does not exist"
Oct 30 04:09:50.525939 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.525868    4960 reconciler_common.go:299] "Volume detached for volume \"pvc-35cf2c66-f433-429c-932a-bb5f878f69a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"csi-f5257c3efa9a6c12782ddbc06dda04660bd20ea308d42e7920bd1a2ecc0f0594\""
Oct 30 04:09:50.542288 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.542237    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs" podStartSLOduration=1.5422215540000002 podStartE2EDuration="1.542221554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:09:50.542044686 +0000 UTC m=+5271.191093125" watchObservedRunningTime="2026-10-30 04:09:50.542221554 +0000 UTC m=+5271.191269973"
Oct 30 04:09:50.580028 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.580000    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"]
Oct 30 04:09:50.583732 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:50.583711    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"]
Oct 30 04:09:51.299152 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.299068    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2"]
Oct 30 04:09:51.299652 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.299631    4960 cpu_manager.go:414] "containerMap: removing container" podUID="a91f7d11-3482-4103-a72b-570a1c8f5019" containerName="loki-compactor"
Oct 30 04:09:51.299729 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.299654    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a91f7d11-3482-4103-a72b-570a1c8f5019" containerName="loki-compactor"
Oct 30 04:09:51.299791 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.299781    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="a91f7d11-3482-4103-a72b-570a1c8f5019" containerName="loki-compactor"
Oct 30 04:09:51.304224 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.304205    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2"
Oct 30 04:09:51.321972 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.321948    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2"]
Oct 30 04:09:51.432154 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.432128    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otlp-tls-certs\") pod \"data-science-usage-logs-collector-57cc577bb6-5dsx2\" (UID: \"8aba5cb8-773c-49bc-827e-6f6c3e27bea2\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2"
Oct 30 04:09:51.432297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.432167    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otc-internal\") pod \"data-science-usage-logs-collector-57cc577bb6-5dsx2\" (UID: \"8aba5cb8-773c-49bc-827e-6f6c3e27bea2\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2"
Oct 30 04:09:51.432378 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.432304    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drd4b\" (UniqueName: \"kubernetes.io/projected/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-kube-api-access-drd4b\") pod \"data-science-usage-logs-collector-57cc577bb6-5dsx2\" (UID: \"8aba5cb8-773c-49bc-827e-6f6c3e27bea2\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2"
Oct 30 04:09:51.501797 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.501773    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a91f7d11-3482-4103-a72b-570a1c8f5019" path="/var/lib/kubelet/pods/a91f7d11-3482-4103-a72b-570a1c8f5019/volumes"
Oct 30 04:09:51.532771 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.532748    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otlp-tls-certs\") pod \"data-science-usage-logs-collector-57cc577bb6-5dsx2\" (UID: \"8aba5cb8-773c-49bc-827e-6f6c3e27bea2\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2"
Oct 30 04:09:51.532886 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.532780    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otc-internal\") pod \"data-science-usage-logs-collector-57cc577bb6-5dsx2\" (UID: \"8aba5cb8-773c-49bc-827e-6f6c3e27bea2\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2"
Oct 30 04:09:51.532994 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.532969    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drd4b\" (UniqueName: \"kubernetes.io/projected/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-kube-api-access-drd4b\") pod \"data-science-usage-logs-collector-57cc577bb6-5dsx2\" (UID: \"8aba5cb8-773c-49bc-827e-6f6c3e27bea2\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2"
Oct 30 04:09:51.533441 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.533418    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otc-internal\") pod \"data-science-usage-logs-collector-57cc577bb6-5dsx2\" (UID: \"8aba5cb8-773c-49bc-827e-6f6c3e27bea2\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2"
Oct 30 04:09:51.535471 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.535452    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otlp-tls-certs\") pod \"data-science-usage-logs-collector-57cc577bb6-5dsx2\" (UID: \"8aba5cb8-773c-49bc-827e-6f6c3e27bea2\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2"
Oct 30 04:09:51.556860 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.556807    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drd4b\" (UniqueName: \"kubernetes.io/projected/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-kube-api-access-drd4b\") pod \"data-science-usage-logs-collector-57cc577bb6-5dsx2\" (UID: \"8aba5cb8-773c-49bc-827e-6f6c3e27bea2\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2"
Oct 30 04:09:51.616237 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.616216    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2"
Oct 30 04:09:51.909873 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.909847    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2"]
Oct 30 04:09:51.910870 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:09:51.910844    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aba5cb8_773c_49bc_827e_6f6c3e27bea2.slice/crio-2d7f00457714c4a7e6746bb20e05df558234ab86f4d2f4ef91e83e00c79832da WatchSource:0}: Error finding container 2d7f00457714c4a7e6746bb20e05df558234ab86f4d2f4ef91e83e00c79832da: Status 404 returned error can't find the container with id 2d7f00457714c4a7e6746bb20e05df558234ab86f4d2f4ef91e83e00c79832da
Oct 30 04:09:51.913107 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.913074    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:09:51.913222 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:51.913138    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:09:52.500030 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:52.499995    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2" event={"ID":"8aba5cb8-773c-49bc-827e-6f6c3e27bea2","Type":"ContainerStarted","Data":"f4bd372af762abb97f35dbb3fc0fc9f886fe806d08cc25ddcf27677140bd05a8"}
Oct 30 04:09:52.500432 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:52.500043    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2" event={"ID":"8aba5cb8-773c-49bc-827e-6f6c3e27bea2","Type":"ContainerStarted","Data":"2d7f00457714c4a7e6746bb20e05df558234ab86f4d2f4ef91e83e00c79832da"}
Oct 30 04:09:52.545969 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:52.545898    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2" podStartSLOduration=1.5458610799999999 podStartE2EDuration="1.54586108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:09:52.54511913 +0000 UTC m=+5273.194167602" watchObservedRunningTime="2026-10-30 04:09:52.54586108 +0000 UTC m=+5273.194909501"
Oct 30 04:09:52.577641 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:52.577609    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz"]
Oct 30 04:09:52.577876 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:52.577839    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz" podUID="7c720a2d-5aef-4479-b7f2-ce543d96bfea" containerName="otc-container" containerID="cri-o://582178cea61625bd34a5314fb0dc16e4ed6d98ca8623e6cdb63abdfeb1917cc2" gracePeriod=30
Oct 30 04:09:52.638459 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:52.638425    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:52.784929 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:52.784364    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"]
Oct 30 04:09:52.784929 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:52.784595    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="647414b0-c92a-445a-8451-8d7272daac7a" containerName="loki-ingester" containerID="cri-o://bd42531b69165c218b6fd1587d1e8692a12a324a3051c084f88ea160bdef5da5" gracePeriod=600
Oct 30 04:09:52.802268 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:52.799993    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"]
Oct 30 04:09:52.802268 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:52.800227    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs" podUID="94a8bb9f-2c6e-42e6-b692-760ef1aee647" containerName="loki-querier" containerID="cri-o://55aed86839419e4d2b96e1865572ee21ecf861b2f964218d285c35021af92e18" gracePeriod=30
Oct 30 04:09:52.821969 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:52.821938    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"]
Oct 30 04:09:52.822190 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:52.822150    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4" podUID="a0fc7b99-6a35-40a6-840b-8d8e578474be" containerName="loki-query-frontend" containerID="cri-o://e16a7f2d4e64b5adf9ce424b563fe9827c8440ffa74f6382b82b6a1ebba2ebec" gracePeriod=30
Oct 30 04:09:52.860398 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:52.860373    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"]
Oct 30 04:09:52.860631 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:52.860608    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj" podUID="48a4dd86-7b31-4634-9531-30286c829729" containerName="gateway" containerID="cri-o://eb8b7362f46fab949aa799a4b8ad1be1c7989e3a910a69e0447bb866ba88d675" gracePeriod=30
Oct 30 04:09:52.860712 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:52.860637    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj" podUID="48a4dd86-7b31-4634-9531-30286c829729" containerName="opa" containerID="cri-o://7cd8a5d433616bc4c7cefcda80e86ac94e870fa1e6c624bd1a02ddb8b22ed46c" gracePeriod=30
Oct 30 04:09:52.863451 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:52.863428    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:52.959122 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:52.959094    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2"]
Oct 30 04:09:52.975940 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:52.975913    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz"
Oct 30 04:09:53.013023 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.012993    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:53.046531 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.046451    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c720a2d-5aef-4479-b7f2-ce543d96bfea-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c720a2d-5aef-4479-b7f2-ce543d96bfea-otlp-tls-certs\") pod \"7c720a2d-5aef-4479-b7f2-ce543d96bfea\" (UID: \"7c720a2d-5aef-4479-b7f2-ce543d96bfea\") "
Oct 30 04:09:53.046531 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.046521    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c720a2d-5aef-4479-b7f2-ce543d96bfea-kube-api-access-lv8vp\" (UniqueName: \"kubernetes.io/projected/7c720a2d-5aef-4479-b7f2-ce543d96bfea-kube-api-access-lv8vp\") pod \"7c720a2d-5aef-4479-b7f2-ce543d96bfea\" (UID: \"7c720a2d-5aef-4479-b7f2-ce543d96bfea\") "
Oct 30 04:09:53.046781 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.046615    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c720a2d-5aef-4479-b7f2-ce543d96bfea-otc-internal\" (UniqueName: \"kubernetes.io/configmap/7c720a2d-5aef-4479-b7f2-ce543d96bfea-otc-internal\") pod \"7c720a2d-5aef-4479-b7f2-ce543d96bfea\" (UID: \"7c720a2d-5aef-4479-b7f2-ce543d96bfea\") "
Oct 30 04:09:53.047261 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.047228    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c720a2d-5aef-4479-b7f2-ce543d96bfea-otc-internal" pod "7c720a2d-5aef-4479-b7f2-ce543d96bfea" (UID: "7c720a2d-5aef-4479-b7f2-ce543d96bfea"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:53.049662 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.049632    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c720a2d-5aef-4479-b7f2-ce543d96bfea-kube-api-access-lv8vp" pod "7c720a2d-5aef-4479-b7f2-ce543d96bfea" (UID: "7c720a2d-5aef-4479-b7f2-ce543d96bfea"). InnerVolumeSpecName "kube-api-access-lv8vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:53.049770 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.049696    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c720a2d-5aef-4479-b7f2-ce543d96bfea-otlp-tls-certs" pod "7c720a2d-5aef-4479-b7f2-ce543d96bfea" (UID: "7c720a2d-5aef-4479-b7f2-ce543d96bfea"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:53.130591 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.130563    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:53.138720 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.138697    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 04:09:53.147800 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.147783    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7c720a2d-5aef-4479-b7f2-ce543d96bfea-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.147879 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.147803    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c720a2d-5aef-4479-b7f2-ce543d96bfea-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.147879 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.147812    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv8vp\" (UniqueName: \"kubernetes.io/projected/7c720a2d-5aef-4479-b7f2-ce543d96bfea-kube-api-access-lv8vp\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.248882 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.248842    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-ca-bundle\") pod \"a0fc7b99-6a35-40a6-840b-8d8e578474be\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") "
Oct 30 04:09:53.249048 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.248928    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-query-frontend-grpc\") pod \"a0fc7b99-6a35-40a6-840b-8d8e578474be\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") "
Oct 30 04:09:53.249048 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.249000    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0fc7b99-6a35-40a6-840b-8d8e578474be-config\" (UniqueName: \"kubernetes.io/configmap/a0fc7b99-6a35-40a6-840b-8d8e578474be-config\") pod \"a0fc7b99-6a35-40a6-840b-8d8e578474be\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") "
Oct 30 04:09:53.249048 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.249030    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-query-frontend-http\") pod \"a0fc7b99-6a35-40a6-840b-8d8e578474be\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") "
Oct 30 04:09:53.249223 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.249092    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0fc7b99-6a35-40a6-840b-8d8e578474be-kube-api-access-l9fs5\" (UniqueName: \"kubernetes.io/projected/a0fc7b99-6a35-40a6-840b-8d8e578474be-kube-api-access-l9fs5\") pod \"a0fc7b99-6a35-40a6-840b-8d8e578474be\" (UID: \"a0fc7b99-6a35-40a6-840b-8d8e578474be\") "
Oct 30 04:09:53.249506 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.249441    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-ca-bundle" pod "a0fc7b99-6a35-40a6-840b-8d8e578474be" (UID: "a0fc7b99-6a35-40a6-840b-8d8e578474be"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:53.249619 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.249514    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0fc7b99-6a35-40a6-840b-8d8e578474be-config" pod "a0fc7b99-6a35-40a6-840b-8d8e578474be" (UID: "a0fc7b99-6a35-40a6-840b-8d8e578474be"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:53.249678 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.249663    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.249728 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.249681    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a0fc7b99-6a35-40a6-840b-8d8e578474be-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.251489 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.251448    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-query-frontend-http" pod "a0fc7b99-6a35-40a6-840b-8d8e578474be" (UID: "a0fc7b99-6a35-40a6-840b-8d8e578474be"). InnerVolumeSpecName "data-science-lokistack-query-frontend-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:53.251981 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.251960    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0fc7b99-6a35-40a6-840b-8d8e578474be-kube-api-access-l9fs5" pod "a0fc7b99-6a35-40a6-840b-8d8e578474be" (UID: "a0fc7b99-6a35-40a6-840b-8d8e578474be"). InnerVolumeSpecName "kube-api-access-l9fs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:53.251981 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.251968    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-query-frontend-grpc" pod "a0fc7b99-6a35-40a6-840b-8d8e578474be" (UID: "a0fc7b99-6a35-40a6-840b-8d8e578474be"). InnerVolumeSpecName "data-science-lokistack-query-frontend-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:53.349014 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.348990    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:53.350116 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.350099    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-query-frontend-http\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.350174 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.350122    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9fs5\" (UniqueName: \"kubernetes.io/projected/a0fc7b99-6a35-40a6-840b-8d8e578474be-kube-api-access-l9fs5\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.350174 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.350134    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/a0fc7b99-6a35-40a6-840b-8d8e578474be-data-science-lokistack-query-frontend-grpc\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.450533 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.450495    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-querier-grpc\") pod \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") "
Oct 30 04:09:53.450663 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.450566    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a8bb9f-2c6e-42e6-b692-760ef1aee647-config\" (UniqueName: \"kubernetes.io/configmap/94a8bb9f-2c6e-42e6-b692-760ef1aee647-config\") pod \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") "
Oct 30 04:09:53.450663 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.450635    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-querier-http\") pod \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") "
Oct 30 04:09:53.450769 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.450669    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-test-loki-lifecycle-secret\") pod \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") "
Oct 30 04:09:53.450769 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.450743    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-ca-bundle\") pod \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") "
Oct 30 04:09:53.450853 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.450773    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a8bb9f-2c6e-42e6-b692-760ef1aee647-kube-api-access-z7bnd\" (UniqueName: \"kubernetes.io/projected/94a8bb9f-2c6e-42e6-b692-760ef1aee647-kube-api-access-z7bnd\") pod \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\" (UID: \"94a8bb9f-2c6e-42e6-b692-760ef1aee647\") "
Oct 30 04:09:53.451132 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.451101    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a8bb9f-2c6e-42e6-b692-760ef1aee647-config" pod "94a8bb9f-2c6e-42e6-b692-760ef1aee647" (UID: "94a8bb9f-2c6e-42e6-b692-760ef1aee647"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:53.451272 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.451174    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-ca-bundle" pod "94a8bb9f-2c6e-42e6-b692-760ef1aee647" (UID: "94a8bb9f-2c6e-42e6-b692-760ef1aee647"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:53.452995 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.452966    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-querier-grpc" pod "94a8bb9f-2c6e-42e6-b692-760ef1aee647" (UID: "94a8bb9f-2c6e-42e6-b692-760ef1aee647"). InnerVolumeSpecName "data-science-lokistack-querier-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:53.452995 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.452982    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-querier-http" pod "94a8bb9f-2c6e-42e6-b692-760ef1aee647" (UID: "94a8bb9f-2c6e-42e6-b692-760ef1aee647"). InnerVolumeSpecName "data-science-lokistack-querier-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:53.453483 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.453465    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a8bb9f-2c6e-42e6-b692-760ef1aee647-kube-api-access-z7bnd" pod "94a8bb9f-2c6e-42e6-b692-760ef1aee647" (UID: "94a8bb9f-2c6e-42e6-b692-760ef1aee647"). InnerVolumeSpecName "kube-api-access-z7bnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:53.453566 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.453464    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-test-loki-lifecycle-secret" pod "94a8bb9f-2c6e-42e6-b692-760ef1aee647" (UID: "94a8bb9f-2c6e-42e6-b692-760ef1aee647"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:53.507143 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.507108    4960 generic.go:356] "Generic (PLEG): container finished" podID="48a4dd86-7b31-4634-9531-30286c829729" containerID="7cd8a5d433616bc4c7cefcda80e86ac94e870fa1e6c624bd1a02ddb8b22ed46c" exitCode=0
Oct 30 04:09:53.507579 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.507205    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj" event={"ID":"48a4dd86-7b31-4634-9531-30286c829729","Type":"ContainerDied","Data":"7cd8a5d433616bc4c7cefcda80e86ac94e870fa1e6c624bd1a02ddb8b22ed46c"}
Oct 30 04:09:53.508581 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.508560    4960 generic.go:356] "Generic (PLEG): container finished" podID="a0fc7b99-6a35-40a6-840b-8d8e578474be" containerID="e16a7f2d4e64b5adf9ce424b563fe9827c8440ffa74f6382b82b6a1ebba2ebec" exitCode=0
Oct 30 04:09:53.508700 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.508619    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"
Oct 30 04:09:53.508700 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.508649    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4" event={"ID":"a0fc7b99-6a35-40a6-840b-8d8e578474be","Type":"ContainerDied","Data":"e16a7f2d4e64b5adf9ce424b563fe9827c8440ffa74f6382b82b6a1ebba2ebec"}
Oct 30 04:09:53.508700 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.508687    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4" event={"ID":"a0fc7b99-6a35-40a6-840b-8d8e578474be","Type":"ContainerDied","Data":"9b5b7c8750747ff67f8fcc264b94976c1bd58717db418f9b3740fead9ea5f047"}
Oct 30 04:09:53.508873 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.508720    4960 scope.go:122] "RemoveContainer" containerID="e16a7f2d4e64b5adf9ce424b563fe9827c8440ffa74f6382b82b6a1ebba2ebec"
Oct 30 04:09:53.510879 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.510641    4960 generic.go:356] "Generic (PLEG): container finished" podID="7c720a2d-5aef-4479-b7f2-ce543d96bfea" containerID="582178cea61625bd34a5314fb0dc16e4ed6d98ca8623e6cdb63abdfeb1917cc2" exitCode=0
Oct 30 04:09:53.510879 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.510681    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz"
Oct 30 04:09:53.510879 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.510735    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz" event={"ID":"7c720a2d-5aef-4479-b7f2-ce543d96bfea","Type":"ContainerDied","Data":"582178cea61625bd34a5314fb0dc16e4ed6d98ca8623e6cdb63abdfeb1917cc2"}
Oct 30 04:09:53.510879 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.510779    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz" event={"ID":"7c720a2d-5aef-4479-b7f2-ce543d96bfea","Type":"ContainerDied","Data":"8ede93cf89e0a222f9ee117b1137de0068c6b05efb1917be9f3f5710786e2fdc"}
Oct 30 04:09:53.515821 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.512649    4960 generic.go:356] "Generic (PLEG): container finished" podID="94a8bb9f-2c6e-42e6-b692-760ef1aee647" containerID="55aed86839419e4d2b96e1865572ee21ecf861b2f964218d285c35021af92e18" exitCode=0
Oct 30 04:09:53.515821 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.512876    4960 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2" secret="" err="secret \"data-science-usage-logs-collector-dockercfg-4t4c9\" not found"
Oct 30 04:09:53.515821 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.513287    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"
Oct 30 04:09:53.515821 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.514384    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs" event={"ID":"94a8bb9f-2c6e-42e6-b692-760ef1aee647","Type":"ContainerDied","Data":"55aed86839419e4d2b96e1865572ee21ecf861b2f964218d285c35021af92e18"}
Oct 30 04:09:53.515821 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.514423    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs" event={"ID":"94a8bb9f-2c6e-42e6-b692-760ef1aee647","Type":"ContainerDied","Data":"49fd4cec9fc8f9f7b9507b2062dbb57366715d9fe1a8c334ccaaf844668cc78b"}
Oct 30 04:09:53.521507 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.521484    4960 scope.go:122] "RemoveContainer" containerID="e16a7f2d4e64b5adf9ce424b563fe9827c8440ffa74f6382b82b6a1ebba2ebec"
Oct 30 04:09:53.522147 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:53.522121    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e16a7f2d4e64b5adf9ce424b563fe9827c8440ffa74f6382b82b6a1ebba2ebec\": container with ID starting with e16a7f2d4e64b5adf9ce424b563fe9827c8440ffa74f6382b82b6a1ebba2ebec not found: ID does not exist" containerID="e16a7f2d4e64b5adf9ce424b563fe9827c8440ffa74f6382b82b6a1ebba2ebec"
Oct 30 04:09:53.522260 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.522154    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e16a7f2d4e64b5adf9ce424b563fe9827c8440ffa74f6382b82b6a1ebba2ebec"} err="failed to get container status \"e16a7f2d4e64b5adf9ce424b563fe9827c8440ffa74f6382b82b6a1ebba2ebec\": rpc error: code = NotFound desc = could not find container \"e16a7f2d4e64b5adf9ce424b563fe9827c8440ffa74f6382b82b6a1ebba2ebec\": container with ID starting with e16a7f2d4e64b5adf9ce424b563fe9827c8440ffa74f6382b82b6a1ebba2ebec not found: ID does not exist"
Oct 30 04:09:53.522260 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.522211    4960 scope.go:122] "RemoveContainer" containerID="582178cea61625bd34a5314fb0dc16e4ed6d98ca8623e6cdb63abdfeb1917cc2"
Oct 30 04:09:53.535470 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.535448    4960 scope.go:122] "RemoveContainer" containerID="582178cea61625bd34a5314fb0dc16e4ed6d98ca8623e6cdb63abdfeb1917cc2"
Oct 30 04:09:53.535751 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:53.535728    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"582178cea61625bd34a5314fb0dc16e4ed6d98ca8623e6cdb63abdfeb1917cc2\": container with ID starting with 582178cea61625bd34a5314fb0dc16e4ed6d98ca8623e6cdb63abdfeb1917cc2 not found: ID does not exist" containerID="582178cea61625bd34a5314fb0dc16e4ed6d98ca8623e6cdb63abdfeb1917cc2"
Oct 30 04:09:53.535828 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.535764    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582178cea61625bd34a5314fb0dc16e4ed6d98ca8623e6cdb63abdfeb1917cc2"} err="failed to get container status \"582178cea61625bd34a5314fb0dc16e4ed6d98ca8623e6cdb63abdfeb1917cc2\": rpc error: code = NotFound desc = could not find container \"582178cea61625bd34a5314fb0dc16e4ed6d98ca8623e6cdb63abdfeb1917cc2\": container with ID starting with 582178cea61625bd34a5314fb0dc16e4ed6d98ca8623e6cdb63abdfeb1917cc2 not found: ID does not exist"
Oct 30 04:09:53.535828 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.535782    4960 scope.go:122] "RemoveContainer" containerID="55aed86839419e4d2b96e1865572ee21ecf861b2f964218d285c35021af92e18"
Oct 30 04:09:53.544111 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.544094    4960 scope.go:122] "RemoveContainer" containerID="55aed86839419e4d2b96e1865572ee21ecf861b2f964218d285c35021af92e18"
Oct 30 04:09:53.544403 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:53.544382    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55aed86839419e4d2b96e1865572ee21ecf861b2f964218d285c35021af92e18\": container with ID starting with 55aed86839419e4d2b96e1865572ee21ecf861b2f964218d285c35021af92e18 not found: ID does not exist" containerID="55aed86839419e4d2b96e1865572ee21ecf861b2f964218d285c35021af92e18"
Oct 30 04:09:53.544502 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.544417    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55aed86839419e4d2b96e1865572ee21ecf861b2f964218d285c35021af92e18"} err="failed to get container status \"55aed86839419e4d2b96e1865572ee21ecf861b2f964218d285c35021af92e18\": rpc error: code = NotFound desc = could not find container \"55aed86839419e4d2b96e1865572ee21ecf861b2f964218d285c35021af92e18\": container with ID starting with 55aed86839419e4d2b96e1865572ee21ecf861b2f964218d285c35021af92e18 not found: ID does not exist"
Oct 30 04:09:53.552002 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.551981    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-querier-http\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.552097 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.552007    4960 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-test-loki-lifecycle-secret\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.552097 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.552022    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.552097 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:53.552029    4960 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-usage-logs-collector-60155212: configmap "data-science-usage-logs-collector-60155212" not found
Oct 30 04:09:53.552097 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.552036    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7bnd\" (UniqueName: \"kubernetes.io/projected/94a8bb9f-2c6e-42e6-b692-760ef1aee647-kube-api-access-z7bnd\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.552097 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.552065    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/94a8bb9f-2c6e-42e6-b692-760ef1aee647-data-science-lokistack-querier-grpc\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.552097 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:53.552033    4960 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-usage-logs-collector-headless-tls: secret "data-science-usage-logs-collector-headless-tls" not found
Oct 30 04:09:53.552097 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:53.552083    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otc-internal podName:8aba5cb8-773c-49bc-827e-6f6c3e27bea2 nodeName:}" failed. No retries permitted until 2026-10-30 04:09:54.052069016 +0000 UTC m=+5274.701117415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otc-internal") pod "data-science-usage-logs-collector-57cc577bb6-5dsx2" (UID: "8aba5cb8-773c-49bc-827e-6f6c3e27bea2") : configmap "data-science-usage-logs-collector-60155212" not found
Oct 30 04:09:53.552471 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.552102    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94a8bb9f-2c6e-42e6-b692-760ef1aee647-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.552471 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:53.552122    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otlp-tls-certs podName:8aba5cb8-773c-49bc-827e-6f6c3e27bea2 nodeName:}" failed. No retries permitted until 2026-10-30 04:09:54.052104534 +0000 UTC m=+5274.701152948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otlp-tls-certs") pod "data-science-usage-logs-collector-57cc577bb6-5dsx2" (UID: "8aba5cb8-773c-49bc-827e-6f6c3e27bea2") : secret "data-science-usage-logs-collector-headless-tls" not found
Oct 30 04:09:53.595243 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.595173    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"]
Oct 30 04:09:53.598472 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.598452    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-m8gc4"]
Oct 30 04:09:53.635574 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.635546    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz"]
Oct 30 04:09:53.637790 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.637767    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-cz8bz"]
Oct 30 04:09:53.674573 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.674544    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"]
Oct 30 04:09:53.675841 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.675820    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-dj7xs"]
Oct 30 04:09:53.739639 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.739618    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:53.854385 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.854312    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ingester-http\") pod \"647414b0-c92a-445a-8451-8d7272daac7a\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") "
Oct 30 04:09:53.854385 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.854364    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-test-loki-shared-secret\") pod \"647414b0-c92a-445a-8451-8d7272daac7a\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") "
Oct 30 04:09:53.854589 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.854435    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ca-bundle\") pod \"647414b0-c92a-445a-8451-8d7272daac7a\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") "
Oct 30 04:09:53.854589 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.854456    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/647414b0-c92a-445a-8451-8d7272daac7a-config\" (UniqueName: \"kubernetes.io/configmap/647414b0-c92a-445a-8451-8d7272daac7a-config\") pod \"647414b0-c92a-445a-8451-8d7272daac7a\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") "
Oct 30 04:09:53.854589 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.854571    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ingester-grpc\") pod \"647414b0-c92a-445a-8451-8d7272daac7a\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") "
Oct 30 04:09:53.854761 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.854711    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") pod \"647414b0-c92a-445a-8451-8d7272daac7a\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") "
Oct 30 04:09:53.854828 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.854763    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/647414b0-c92a-445a-8451-8d7272daac7a-kube-api-access-cwhwl\" (UniqueName: \"kubernetes.io/projected/647414b0-c92a-445a-8451-8d7272daac7a-kube-api-access-cwhwl\") pod \"647414b0-c92a-445a-8451-8d7272daac7a\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") "
Oct 30 04:09:53.854879 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.854838    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") pod \"647414b0-c92a-445a-8451-8d7272daac7a\" (UID: \"647414b0-c92a-445a-8451-8d7272daac7a\") "
Oct 30 04:09:53.854930 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.854878    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647414b0-c92a-445a-8451-8d7272daac7a-config" pod "647414b0-c92a-445a-8451-8d7272daac7a" (UID: "647414b0-c92a-445a-8451-8d7272daac7a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:53.854930 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.854888    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ca-bundle" pod "647414b0-c92a-445a-8451-8d7272daac7a" (UID: "647414b0-c92a-445a-8451-8d7272daac7a"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:53.855256 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.855153    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.855256 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.855198    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/647414b0-c92a-445a-8451-8d7272daac7a-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.857723 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.857694    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ingester-http" pod "647414b0-c92a-445a-8451-8d7272daac7a" (UID: "647414b0-c92a-445a-8451-8d7272daac7a"). InnerVolumeSpecName "data-science-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:53.857983 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.857958    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647414b0-c92a-445a-8451-8d7272daac7a-kube-api-access-cwhwl" pod "647414b0-c92a-445a-8451-8d7272daac7a" (UID: "647414b0-c92a-445a-8451-8d7272daac7a"). InnerVolumeSpecName "kube-api-access-cwhwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:53.858129 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.858109    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ingester-grpc" pod "647414b0-c92a-445a-8451-8d7272daac7a" (UID: "647414b0-c92a-445a-8451-8d7272daac7a"). InnerVolumeSpecName "data-science-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:53.858212 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.858153    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-test-loki-shared-secret" pod "647414b0-c92a-445a-8451-8d7272daac7a" (UID: "647414b0-c92a-445a-8451-8d7272daac7a"). InnerVolumeSpecName "test-loki-shared-secret". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:53.858567 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.858547    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de" pod "647414b0-c92a-445a-8451-8d7272daac7a" (UID: "647414b0-c92a-445a-8451-8d7272daac7a"). InnerVolumeSpecName "pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:09:53.858567 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.858558    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364" pod "647414b0-c92a-445a-8451-8d7272daac7a" (UID: "647414b0-c92a-445a-8451-8d7272daac7a"). InnerVolumeSpecName "pvc-4575788d-4b47-476e-89a1-a05676f07b7e". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:09:53.955194 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:53.955138    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:09:53.955577 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.955561    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ingester-grpc\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.955645 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.955595    4960 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") on node \"ip-10-0-128-241.ec2.internal\" "
Oct 30 04:09:53.955645 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.955605    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwhwl\" (UniqueName: \"kubernetes.io/projected/647414b0-c92a-445a-8451-8d7272daac7a-kube-api-access-cwhwl\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.955645 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.955619    4960 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4575788d-4b47-476e-89a1-a05676f07b7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") on node \"ip-10-0-128-241.ec2.internal\" "
Oct 30 04:09:53.955645 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.955628    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-data-science-lokistack-ingester-http\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.955645 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.955637    4960 reconciler_common.go:299] "Volume detached for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/647414b0-c92a-445a-8451-8d7272daac7a-test-loki-shared-secret\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.966148 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.966132    4960 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4575788d-4b47-476e-89a1-a05676f07b7e" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364") on node "ip-10-0-128-241.ec2.internal"
Oct 30 04:09:53.969659 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:53.969641    4960 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de") on node "ip-10-0-128-241.ec2.internal"
Oct 30 04:09:54.056511 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.056486    4960 reconciler_common.go:299] "Volume detached for volume \"pvc-4575788d-4b47-476e-89a1-a05676f07b7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"csi-07e2e97f2246d642a8e47279f26d69dde46412f7218628a5c9d916389d78b478\""
Oct 30 04:09:54.056511 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.056511    4960 reconciler_common.go:299] "Volume detached for volume \"pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"csi-79b8ce0216485321c872c3dd9c6696ead485e08c1d139e467da9818fbf093e5a\""
Oct 30 04:09:54.056682 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:54.056548    4960 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-usage-logs-collector-headless-tls: secret "data-science-usage-logs-collector-headless-tls" not found
Oct 30 04:09:54.056682 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:54.056564    4960 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-usage-logs-collector-60155212: configmap "data-science-usage-logs-collector-60155212" not found
Oct 30 04:09:54.056682 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:54.056611    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otlp-tls-certs podName:8aba5cb8-773c-49bc-827e-6f6c3e27bea2 nodeName:}" failed. No retries permitted until 2026-10-30 04:09:55.056595382 +0000 UTC m=+5275.705643785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otlp-tls-certs") pod "data-science-usage-logs-collector-57cc577bb6-5dsx2" (UID: "8aba5cb8-773c-49bc-827e-6f6c3e27bea2") : secret "data-science-usage-logs-collector-headless-tls" not found
Oct 30 04:09:54.056682 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:54.056625    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otc-internal podName:8aba5cb8-773c-49bc-827e-6f6c3e27bea2 nodeName:}" failed. No retries permitted until 2026-10-30 04:09:55.056618914 +0000 UTC m=+5275.705667316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otc-internal") pod "data-science-usage-logs-collector-57cc577bb6-5dsx2" (UID: "8aba5cb8-773c-49bc-827e-6f6c3e27bea2") : configmap "data-science-usage-logs-collector-60155212" not found
Oct 30 04:09:54.120963 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.120939    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 04:09:54.258828 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.258799    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-tls-secret\" (UniqueName: \"kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-tls-secret\") pod \"48a4dd86-7b31-4634-9531-30286c829729\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") "
Oct 30 04:09:54.258974 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.258838    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-kube-api-access-tm2z2\" (UniqueName: \"kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-kube-api-access-tm2z2\") pod \"48a4dd86-7b31-4634-9531-30286c829729\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") "
Oct 30 04:09:54.258974 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.258880    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-ca-bundle\") pod \"48a4dd86-7b31-4634-9531-30286c829729\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") "
Oct 30 04:09:54.259059 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.258986    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-gateway-ca-bundle\") pod \"48a4dd86-7b31-4634-9531-30286c829729\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") "
Oct 30 04:09:54.259094 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.259069    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-gateway-client-http\") pod \"48a4dd86-7b31-4634-9531-30286c829729\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") "
Oct 30 04:09:54.259141 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.259097    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48a4dd86-7b31-4634-9531-30286c829729-tenants\" (UniqueName: \"kubernetes.io/secret/48a4dd86-7b31-4634-9531-30286c829729-tenants\") pod \"48a4dd86-7b31-4634-9531-30286c829729\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") "
Oct 30 04:09:54.259141 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.259126    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-lokistack-gateway\") pod \"48a4dd86-7b31-4634-9531-30286c829729\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") "
Oct 30 04:09:54.259269 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.259155    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-rbac\" (UniqueName: \"kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-rbac\") pod \"48a4dd86-7b31-4634-9531-30286c829729\" (UID: \"48a4dd86-7b31-4634-9531-30286c829729\") "
Oct 30 04:09:54.259332 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.259286    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-ca-bundle" pod "48a4dd86-7b31-4634-9531-30286c829729" (UID: "48a4dd86-7b31-4634-9531-30286c829729"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:54.259632 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.259528    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.261997 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.261956    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-kube-api-access-tm2z2" pod "48a4dd86-7b31-4634-9531-30286c829729" (UID: "48a4dd86-7b31-4634-9531-30286c829729"). InnerVolumeSpecName "kube-api-access-tm2z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:54.262113 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.261966    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-tls-secret" pod "48a4dd86-7b31-4634-9531-30286c829729" (UID: "48a4dd86-7b31-4634-9531-30286c829729"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:54.262113 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.262065    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-gateway-ca-bundle" pod "48a4dd86-7b31-4634-9531-30286c829729" (UID: "48a4dd86-7b31-4634-9531-30286c829729"). InnerVolumeSpecName "data-science-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:54.262318 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.262229    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-gateway-client-http" pod "48a4dd86-7b31-4634-9531-30286c829729" (UID: "48a4dd86-7b31-4634-9531-30286c829729"). InnerVolumeSpecName "data-science-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:54.264684 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.264666    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-rbac" pod "48a4dd86-7b31-4634-9531-30286c829729" (UID: "48a4dd86-7b31-4634-9531-30286c829729"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:54.265959 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.265943    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-lokistack-gateway" pod "48a4dd86-7b31-4634-9531-30286c829729" (UID: "48a4dd86-7b31-4634-9531-30286c829729"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:54.267684 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.267665    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a4dd86-7b31-4634-9531-30286c829729-tenants" pod "48a4dd86-7b31-4634-9531-30286c829729" (UID: "48a4dd86-7b31-4634-9531-30286c829729"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:54.360484 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.360431    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-gateway-client-http\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.360484 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.360452    4960 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/48a4dd86-7b31-4634-9531-30286c829729-tenants\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.360484 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.360462    4960 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-lokistack-gateway\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.360484 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.360470    4960 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/48a4dd86-7b31-4634-9531-30286c829729-rbac\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.360484 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.360479    4960 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-tls-secret\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.360484 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.360488    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm2z2\" (UniqueName: \"kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-kube-api-access-tm2z2\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.360720 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.360497    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/48a4dd86-7b31-4634-9531-30286c829729-data-science-lokistack-gateway-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.520812 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.520785    4960 generic.go:356] "Generic (PLEG): container finished" podID="647414b0-c92a-445a-8451-8d7272daac7a" containerID="bd42531b69165c218b6fd1587d1e8692a12a324a3051c084f88ea160bdef5da5" exitCode=0
Oct 30 04:09:54.521231 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.520830    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:54.521231 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.520866    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"647414b0-c92a-445a-8451-8d7272daac7a","Type":"ContainerDied","Data":"bd42531b69165c218b6fd1587d1e8692a12a324a3051c084f88ea160bdef5da5"}
Oct 30 04:09:54.521231 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.520911    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"647414b0-c92a-445a-8451-8d7272daac7a","Type":"ContainerDied","Data":"04c2aa0e8cbc4f07e7afca8d7f3511534a032f455949ba59bfc33fa88e201498"}
Oct 30 04:09:54.521231 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.520941    4960 scope.go:122] "RemoveContainer" containerID="bd42531b69165c218b6fd1587d1e8692a12a324a3051c084f88ea160bdef5da5"
Oct 30 04:09:54.522731 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.522664    4960 generic.go:356] "Generic (PLEG): container finished" podID="48a4dd86-7b31-4634-9531-30286c829729" containerID="eb8b7362f46fab949aa799a4b8ad1be1c7989e3a910a69e0447bb866ba88d675" exitCode=0
Oct 30 04:09:54.522731 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.522708    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"
Oct 30 04:09:54.522968 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.522733    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj" event={"ID":"48a4dd86-7b31-4634-9531-30286c829729","Type":"ContainerDied","Data":"eb8b7362f46fab949aa799a4b8ad1be1c7989e3a910a69e0447bb866ba88d675"}
Oct 30 04:09:54.522968 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.522775    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj" event={"ID":"48a4dd86-7b31-4634-9531-30286c829729","Type":"ContainerDied","Data":"123f69b1437003e3cdf12395df325444c85115a84dd1761bd0c101b9fc0f17cf"}
Oct 30 04:09:54.523088 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.522986    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2" podUID="8aba5cb8-773c-49bc-827e-6f6c3e27bea2" containerName="otc-container" containerID="cri-o://f4bd372af762abb97f35dbb3fc0fc9f886fe806d08cc25ddcf27677140bd05a8" gracePeriod=30
Oct 30 04:09:54.530696 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.530667    4960 scope.go:122] "RemoveContainer" containerID="bd42531b69165c218b6fd1587d1e8692a12a324a3051c084f88ea160bdef5da5"
Oct 30 04:09:54.531032 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:54.531012    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd42531b69165c218b6fd1587d1e8692a12a324a3051c084f88ea160bdef5da5\": container with ID starting with bd42531b69165c218b6fd1587d1e8692a12a324a3051c084f88ea160bdef5da5 not found: ID does not exist" containerID="bd42531b69165c218b6fd1587d1e8692a12a324a3051c084f88ea160bdef5da5"
Oct 30 04:09:54.531098 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.531038    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd42531b69165c218b6fd1587d1e8692a12a324a3051c084f88ea160bdef5da5"} err="failed to get container status \"bd42531b69165c218b6fd1587d1e8692a12a324a3051c084f88ea160bdef5da5\": rpc error: code = NotFound desc = could not find container \"bd42531b69165c218b6fd1587d1e8692a12a324a3051c084f88ea160bdef5da5\": container with ID starting with bd42531b69165c218b6fd1587d1e8692a12a324a3051c084f88ea160bdef5da5 not found: ID does not exist"
Oct 30 04:09:54.531098 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.531054    4960 scope.go:122] "RemoveContainer" containerID="7cd8a5d433616bc4c7cefcda80e86ac94e870fa1e6c624bd1a02ddb8b22ed46c"
Oct 30 04:09:54.539113 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.539097    4960 scope.go:122] "RemoveContainer" containerID="eb8b7362f46fab949aa799a4b8ad1be1c7989e3a910a69e0447bb866ba88d675"
Oct 30 04:09:54.575364 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.575344    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"]
Oct 30 04:09:54.588941 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.588920    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"]
Oct 30 04:09:54.614251 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.614196    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"]
Oct 30 04:09:54.617504 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.617485    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-hmmfj"]
Oct 30 04:09:54.634360 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.634342    4960 scope.go:122] "RemoveContainer" containerID="7cd8a5d433616bc4c7cefcda80e86ac94e870fa1e6c624bd1a02ddb8b22ed46c"
Oct 30 04:09:54.634666 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:54.634646    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd8a5d433616bc4c7cefcda80e86ac94e870fa1e6c624bd1a02ddb8b22ed46c\": container with ID starting with 7cd8a5d433616bc4c7cefcda80e86ac94e870fa1e6c624bd1a02ddb8b22ed46c not found: ID does not exist" containerID="7cd8a5d433616bc4c7cefcda80e86ac94e870fa1e6c624bd1a02ddb8b22ed46c"
Oct 30 04:09:54.634768 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.634677    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd8a5d433616bc4c7cefcda80e86ac94e870fa1e6c624bd1a02ddb8b22ed46c"} err="failed to get container status \"7cd8a5d433616bc4c7cefcda80e86ac94e870fa1e6c624bd1a02ddb8b22ed46c\": rpc error: code = NotFound desc = could not find container \"7cd8a5d433616bc4c7cefcda80e86ac94e870fa1e6c624bd1a02ddb8b22ed46c\": container with ID starting with 7cd8a5d433616bc4c7cefcda80e86ac94e870fa1e6c624bd1a02ddb8b22ed46c not found: ID does not exist"
Oct 30 04:09:54.634768 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.634702    4960 scope.go:122] "RemoveContainer" containerID="eb8b7362f46fab949aa799a4b8ad1be1c7989e3a910a69e0447bb866ba88d675"
Oct 30 04:09:54.635055 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:54.635038    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb8b7362f46fab949aa799a4b8ad1be1c7989e3a910a69e0447bb866ba88d675\": container with ID starting with eb8b7362f46fab949aa799a4b8ad1be1c7989e3a910a69e0447bb866ba88d675 not found: ID does not exist" containerID="eb8b7362f46fab949aa799a4b8ad1be1c7989e3a910a69e0447bb866ba88d675"
Oct 30 04:09:54.635110 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.635063    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb8b7362f46fab949aa799a4b8ad1be1c7989e3a910a69e0447bb866ba88d675"} err="failed to get container status \"eb8b7362f46fab949aa799a4b8ad1be1c7989e3a910a69e0447bb866ba88d675\": rpc error: code = NotFound desc = could not find container \"eb8b7362f46fab949aa799a4b8ad1be1c7989e3a910a69e0447bb866ba88d675\": container with ID starting with eb8b7362f46fab949aa799a4b8ad1be1c7989e3a910a69e0447bb866ba88d675 not found: ID does not exist"
Oct 30 04:09:54.769348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.769324    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2"
Oct 30 04:09:54.864206 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.864150    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otc-internal\" (UniqueName: \"kubernetes.io/configmap/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otc-internal\") pod \"8aba5cb8-773c-49bc-827e-6f6c3e27bea2\" (UID: \"8aba5cb8-773c-49bc-827e-6f6c3e27bea2\") "
Oct 30 04:09:54.864387 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.864232    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-kube-api-access-drd4b\" (UniqueName: \"kubernetes.io/projected/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-kube-api-access-drd4b\") pod \"8aba5cb8-773c-49bc-827e-6f6c3e27bea2\" (UID: \"8aba5cb8-773c-49bc-827e-6f6c3e27bea2\") "
Oct 30 04:09:54.864387 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.864300    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otlp-tls-certs\") pod \"8aba5cb8-773c-49bc-827e-6f6c3e27bea2\" (UID: \"8aba5cb8-773c-49bc-827e-6f6c3e27bea2\") "
Oct 30 04:09:54.864552 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.864525    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otc-internal" pod "8aba5cb8-773c-49bc-827e-6f6c3e27bea2" (UID: "8aba5cb8-773c-49bc-827e-6f6c3e27bea2"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:54.866540 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.866509    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otlp-tls-certs" pod "8aba5cb8-773c-49bc-827e-6f6c3e27bea2" (UID: "8aba5cb8-773c-49bc-827e-6f6c3e27bea2"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:54.866540 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.866525    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-kube-api-access-drd4b" pod "8aba5cb8-773c-49bc-827e-6f6c3e27bea2" (UID: "8aba5cb8-773c-49bc-827e-6f6c3e27bea2"). InnerVolumeSpecName "kube-api-access-drd4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:54.965001 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.964972    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.965001 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.964997    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drd4b\" (UniqueName: \"kubernetes.io/projected/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-kube-api-access-drd4b\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.965144 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:54.965006    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8aba5cb8-773c-49bc-827e-6f6c3e27bea2-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:09:55.502418 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:55.502386    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a4dd86-7b31-4634-9531-30286c829729" path="/var/lib/kubelet/pods/48a4dd86-7b31-4634-9531-30286c829729/volumes"
Oct 30 04:09:55.502831 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:55.502818    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="647414b0-c92a-445a-8451-8d7272daac7a" path="/var/lib/kubelet/pods/647414b0-c92a-445a-8451-8d7272daac7a/volumes"
Oct 30 04:09:55.503160 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:55.503149    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c720a2d-5aef-4479-b7f2-ce543d96bfea" path="/var/lib/kubelet/pods/7c720a2d-5aef-4479-b7f2-ce543d96bfea/volumes"
Oct 30 04:09:55.503765 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:55.503742    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a8bb9f-2c6e-42e6-b692-760ef1aee647" path="/var/lib/kubelet/pods/94a8bb9f-2c6e-42e6-b692-760ef1aee647/volumes"
Oct 30 04:09:55.504136 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:55.504122    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0fc7b99-6a35-40a6-840b-8d8e578474be" path="/var/lib/kubelet/pods/a0fc7b99-6a35-40a6-840b-8d8e578474be/volumes"
Oct 30 04:09:55.529560 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:55.529532    4960 generic.go:356] "Generic (PLEG): container finished" podID="8aba5cb8-773c-49bc-827e-6f6c3e27bea2" containerID="f4bd372af762abb97f35dbb3fc0fc9f886fe806d08cc25ddcf27677140bd05a8" exitCode=0
Oct 30 04:09:55.529912 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:55.529573    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2" event={"ID":"8aba5cb8-773c-49bc-827e-6f6c3e27bea2","Type":"ContainerDied","Data":"f4bd372af762abb97f35dbb3fc0fc9f886fe806d08cc25ddcf27677140bd05a8"}
Oct 30 04:09:55.529912 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:55.529607    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2" event={"ID":"8aba5cb8-773c-49bc-827e-6f6c3e27bea2","Type":"ContainerDied","Data":"2d7f00457714c4a7e6746bb20e05df558234ab86f4d2f4ef91e83e00c79832da"}
Oct 30 04:09:55.529912 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:55.529628    4960 scope.go:122] "RemoveContainer" containerID="f4bd372af762abb97f35dbb3fc0fc9f886fe806d08cc25ddcf27677140bd05a8"
Oct 30 04:09:55.529912 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:55.529576    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2"
Oct 30 04:09:55.537939 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:55.537913    4960 scope.go:122] "RemoveContainer" containerID="f4bd372af762abb97f35dbb3fc0fc9f886fe806d08cc25ddcf27677140bd05a8"
Oct 30 04:09:55.538198 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:55.538158    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4bd372af762abb97f35dbb3fc0fc9f886fe806d08cc25ddcf27677140bd05a8\": container with ID starting with f4bd372af762abb97f35dbb3fc0fc9f886fe806d08cc25ddcf27677140bd05a8 not found: ID does not exist" containerID="f4bd372af762abb97f35dbb3fc0fc9f886fe806d08cc25ddcf27677140bd05a8"
Oct 30 04:09:55.538291 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:55.538237    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4bd372af762abb97f35dbb3fc0fc9f886fe806d08cc25ddcf27677140bd05a8"} err="failed to get container status \"f4bd372af762abb97f35dbb3fc0fc9f886fe806d08cc25ddcf27677140bd05a8\": rpc error: code = NotFound desc = could not find container \"f4bd372af762abb97f35dbb3fc0fc9f886fe806d08cc25ddcf27677140bd05a8\": container with ID starting with f4bd372af762abb97f35dbb3fc0fc9f886fe806d08cc25ddcf27677140bd05a8 not found: ID does not exist"
Oct 30 04:09:55.586800 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:55.586780    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2"]
Oct 30 04:09:55.600666 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:55.600645    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-5dsx2"]
Oct 30 04:09:56.940856 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.940826    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"]
Oct 30 04:09:56.941252 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941155    4960 cpu_manager.go:414] "containerMap: removing container" podUID="8aba5cb8-773c-49bc-827e-6f6c3e27bea2" containerName="otc-container"
Oct 30 04:09:56.941252 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941170    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aba5cb8-773c-49bc-827e-6f6c3e27bea2" containerName="otc-container"
Oct 30 04:09:56.941252 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941207    4960 cpu_manager.go:414] "containerMap: removing container" podUID="a0fc7b99-6a35-40a6-840b-8d8e578474be" containerName="loki-query-frontend"
Oct 30 04:09:56.941252 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941213    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0fc7b99-6a35-40a6-840b-8d8e578474be" containerName="loki-query-frontend"
Oct 30 04:09:56.941252 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941221    4960 cpu_manager.go:414] "containerMap: removing container" podUID="48a4dd86-7b31-4634-9531-30286c829729" containerName="gateway"
Oct 30 04:09:56.941252 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941226    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a4dd86-7b31-4634-9531-30286c829729" containerName="gateway"
Oct 30 04:09:56.941252 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941235    4960 cpu_manager.go:414] "containerMap: removing container" podUID="48a4dd86-7b31-4634-9531-30286c829729" containerName="opa"
Oct 30 04:09:56.941252 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941239    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a4dd86-7b31-4634-9531-30286c829729" containerName="opa"
Oct 30 04:09:56.941252 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941249    4960 cpu_manager.go:414] "containerMap: removing container" podUID="7c720a2d-5aef-4479-b7f2-ce543d96bfea" containerName="otc-container"
Oct 30 04:09:56.941252 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941254    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c720a2d-5aef-4479-b7f2-ce543d96bfea" containerName="otc-container"
Oct 30 04:09:56.941550 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941277    4960 cpu_manager.go:414] "containerMap: removing container" podUID="647414b0-c92a-445a-8451-8d7272daac7a" containerName="loki-ingester"
Oct 30 04:09:56.941550 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941283    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647414b0-c92a-445a-8451-8d7272daac7a" containerName="loki-ingester"
Oct 30 04:09:56.941550 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941290    4960 cpu_manager.go:414] "containerMap: removing container" podUID="94a8bb9f-2c6e-42e6-b692-760ef1aee647" containerName="loki-querier"
Oct 30 04:09:56.941550 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941294    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a8bb9f-2c6e-42e6-b692-760ef1aee647" containerName="loki-querier"
Oct 30 04:09:56.941550 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941350    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a4dd86-7b31-4634-9531-30286c829729" containerName="gateway"
Oct 30 04:09:56.941550 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941358    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a4dd86-7b31-4634-9531-30286c829729" containerName="opa"
Oct 30 04:09:56.941550 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941365    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c720a2d-5aef-4479-b7f2-ce543d96bfea" containerName="otc-container"
Oct 30 04:09:56.941550 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941373    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="647414b0-c92a-445a-8451-8d7272daac7a" containerName="loki-ingester"
Oct 30 04:09:56.941550 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941380    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a8bb9f-2c6e-42e6-b692-760ef1aee647" containerName="loki-querier"
Oct 30 04:09:56.941550 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941385    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aba5cb8-773c-49bc-827e-6f6c3e27bea2" containerName="otc-container"
Oct 30 04:09:56.941550 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.941394    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0fc7b99-6a35-40a6-840b-8d8e578474be" containerName="loki-query-frontend"
Oct 30 04:09:56.945601 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.945579    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:09:56.972580 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.972556    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:56.977636 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:56.977612    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"]
Oct 30 04:09:57.039469 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.039442    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-grpc\""
Oct 30 04:09:57.039631 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.039568    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-http\""
Oct 30 04:09:57.039631 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.039615    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ca-bundle\""
Oct 30 04:09:57.039738 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.039690    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-dockercfg-sk4rz\""
Oct 30 04:09:57.039738 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.039726    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:09:57.039837 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.039727    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:09:57.039940 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.039920    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-config\""
Oct 30 04:09:57.081722 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.081696    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-jdtjr\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:09:57.081821 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.081730    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-jdtjr\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:09:57.081821 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.081751    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bpsb\" (UniqueName: \"kubernetes.io/projected/4f6549b0-c354-45d7-a9ba-ec2680ec4372-kube-api-access-9bpsb\") pod \"data-science-lokistack-distributor-58947679b7-jdtjr\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:09:57.081907 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.081815    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-jdtjr\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:09:57.081945 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.081923    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f6549b0-c354-45d7-a9ba-ec2680ec4372-config\") pod \"data-science-lokistack-distributor-58947679b7-jdtjr\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:09:57.098105 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.098053    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:57.117651 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.117628    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"]
Oct 30 04:09:57.121513 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.121495    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.142399 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.142374    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"]
Oct 30 04:09:57.183054 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.183024    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f6549b0-c354-45d7-a9ba-ec2680ec4372-config\") pod \"data-science-lokistack-distributor-58947679b7-jdtjr\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:09:57.183230 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.183070    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-hnwjx\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.183230 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.183093    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2b2s\" (UniqueName: \"kubernetes.io/projected/37e1f157-4854-45fa-81cf-454c7841587a-kube-api-access-w2b2s\") pod \"data-science-lokistack-querier-56ffbd5fdc-hnwjx\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.183230 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.183212    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-jdtjr\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:09:57.183414 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.183271    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-jdtjr\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:09:57.183414 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.183311    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpsb\" (UniqueName: \"kubernetes.io/projected/4f6549b0-c354-45d7-a9ba-ec2680ec4372-kube-api-access-9bpsb\") pod \"data-science-lokistack-distributor-58947679b7-jdtjr\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:09:57.183414 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.183341    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-hnwjx\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.183414 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.183386    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-jdtjr\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:09:57.183414 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.183412    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37e1f157-4854-45fa-81cf-454c7841587a-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-hnwjx\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.183703 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.183437    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-hnwjx\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.183703 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.183509    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-hnwjx\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.183873 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.183853    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-jdtjr\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:09:57.183963 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.183946    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f6549b0-c354-45d7-a9ba-ec2680ec4372-config\") pod \"data-science-lokistack-distributor-58947679b7-jdtjr\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:09:57.186054 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.186029    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-jdtjr\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:09:57.186401 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.186385    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-jdtjr\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:09:57.212077 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.209244    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bpsb\" (UniqueName: \"kubernetes.io/projected/4f6549b0-c354-45d7-a9ba-ec2680ec4372-kube-api-access-9bpsb\") pod \"data-science-lokistack-distributor-58947679b7-jdtjr\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:09:57.225306 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.225276    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:57.239838 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.239817    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-lifecycle-secret\""
Oct 30 04:09:57.239956 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.239887    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-http\""
Oct 30 04:09:57.239956 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.239900    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-grpc\""
Oct 30 04:09:57.257511 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.257487    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:09:57.275832 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.275806    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"]
Oct 30 04:09:57.280842 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.280813    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:09:57.284317 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.284292    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-hnwjx\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.284424 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.284333    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37e1f157-4854-45fa-81cf-454c7841587a-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-hnwjx\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.284424 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.284356    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-hnwjx\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.284424 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.284396    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-hnwjx\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.284969 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.284950    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-hnwjx\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.285376 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.285356    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37e1f157-4854-45fa-81cf-454c7841587a-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-hnwjx\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.285452 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.285432    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-hnwjx\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.285509 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.285479    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2b2s\" (UniqueName: \"kubernetes.io/projected/37e1f157-4854-45fa-81cf-454c7841587a-kube-api-access-w2b2s\") pod \"data-science-lokistack-querier-56ffbd5fdc-hnwjx\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.287542 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.287511    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-hnwjx\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.288525 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.288473    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-hnwjx\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.288908 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.288886    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-hnwjx\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.299052 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.299024    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"]
Oct 30 04:09:57.307841 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.307814    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2b2s\" (UniqueName: \"kubernetes.io/projected/37e1f157-4854-45fa-81cf-454c7841587a-kube-api-access-w2b2s\") pod \"data-science-lokistack-querier-56ffbd5fdc-hnwjx\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.340386 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.340361    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-query-frontend-grpc\""
Oct 30 04:09:57.340386 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.340365    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-query-frontend-http\""
Oct 30 04:09:57.350725 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.350675    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:57.386797 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.386773    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-f746fc66d-xbkk9\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:09:57.386921 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.386817    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-f746fc66d-xbkk9\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:09:57.386921 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.386898    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/89348102-649b-4362-892e-51db15bb9e18-config\") pod \"data-science-lokistack-query-frontend-f746fc66d-xbkk9\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:09:57.387006 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.386951    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvn6x\" (UniqueName: \"kubernetes.io/projected/89348102-649b-4362-892e-51db15bb9e18-kube-api-access-wvn6x\") pod \"data-science-lokistack-query-frontend-f746fc66d-xbkk9\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:09:57.387006 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.386968    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-f746fc66d-xbkk9\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:09:57.433703 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.433673    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:57.484092 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.483969    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:57.489829 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.489801    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-f746fc66d-xbkk9\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:09:57.489962 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.489866    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/89348102-649b-4362-892e-51db15bb9e18-config\") pod \"data-science-lokistack-query-frontend-f746fc66d-xbkk9\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:09:57.489962 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.489902    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvn6x\" (UniqueName: \"kubernetes.io/projected/89348102-649b-4362-892e-51db15bb9e18-kube-api-access-wvn6x\") pod \"data-science-lokistack-query-frontend-f746fc66d-xbkk9\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:09:57.489962 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.489928    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-f746fc66d-xbkk9\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:09:57.490121 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.490007    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-f746fc66d-xbkk9\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:09:57.491154 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.491132    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-f746fc66d-xbkk9\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:09:57.493161 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.493053    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/89348102-649b-4362-892e-51db15bb9e18-config\") pod \"data-science-lokistack-query-frontend-f746fc66d-xbkk9\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:09:57.493325 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.493306    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-f746fc66d-xbkk9\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:09:57.493425 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.493407    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-f746fc66d-xbkk9\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:09:57.506328 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.506303    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aba5cb8-773c-49bc-827e-6f6c3e27bea2" path="/var/lib/kubelet/pods/8aba5cb8-773c-49bc-827e-6f6c3e27bea2/volumes"
Oct 30 04:09:57.511525 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.511498    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvn6x\" (UniqueName: \"kubernetes.io/projected/89348102-649b-4362-892e-51db15bb9e18-kube-api-access-wvn6x\") pod \"data-science-lokistack-query-frontend-f746fc66d-xbkk9\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:09:57.526263 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.526240    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"]
Oct 30 04:09:57.530856 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.530837    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.552694 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.552672    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"]
Oct 30 04:09:57.560035 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.559636    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"]
Oct 30 04:09:57.561089 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.561069    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\""
Oct 30 04:09:57.561261 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.561240    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-ca-bundle\""
Oct 30 04:09:57.561731 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:09:57.561705    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6549b0_c354_45d7_a9ba_ec2680ec4372.slice/crio-550e29d58cdf9bfb313fb85947e3f8c1796c304b8b1786fe7f5aaf5c4241bd1a WatchSource:0}: Error finding container 550e29d58cdf9bfb313fb85947e3f8c1796c304b8b1786fe7f5aaf5c4241bd1a: Status 404 returned error can't find the container with id 550e29d58cdf9bfb313fb85947e3f8c1796c304b8b1786fe7f5aaf5c4241bd1a
Oct 30 04:09:57.591198 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.591155    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpcvc\" (UniqueName: \"kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-kube-api-access-kpcvc\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.591198 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.591197    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-rbac\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.591322 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.591286    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.591363 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.591336    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/4d802189-cb0b-41c2-ba62-e11231925023-tenants\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.591363 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.591358    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-tls-secret\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.591434 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.591383    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-lokistack-gateway\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.591434 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.591404    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.591546 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.591530    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.596938 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.596917    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:09:57.603804 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.603782    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:57.616167 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.616134    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\""
Oct 30 04:09:57.616167 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.616135    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-http\""
Oct 30 04:09:57.616340 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.616148    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-client-http\""
Oct 30 04:09:57.616340 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.616287    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-dockercfg-5mcwk\""
Oct 30 04:09:57.693620 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.692646    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/4d802189-cb0b-41c2-ba62-e11231925023-tenants\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.693620 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.692684    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-tls-secret\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.693620 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.692711    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-lokistack-gateway\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.693620 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.692726    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.693620 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.692760    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.693620 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.692786    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpcvc\" (UniqueName: \"kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-kube-api-access-kpcvc\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.693620 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.692809    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-rbac\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.693620 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.692839    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.693620 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.693376    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.694850 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.694473    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-rbac\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.694850 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:57.694538    4960 projected.go:266] Couldn't get secret redhat-ods-monitoring/data-science-lokistack-gateway-http: secret "data-science-lokistack-gateway-http" not found
Oct 30 04:09:57.694850 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:57.694564    4960 projected.go:266] Couldn't get secret redhat-ods-monitoring/data-science-lokistack-gateway-http: secret "data-science-lokistack-gateway-http" not found
Oct 30 04:09:57.694850 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:57.694577    4960 projected.go:196] Error preparing data for projected volume tls-secret for pod redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml: secret "data-science-lokistack-gateway-http" not found
Oct 30 04:09:57.694850 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:09:57.694632    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-tls-secret podName:4d802189-cb0b-41c2-ba62-e11231925023 nodeName:}" failed. No retries permitted until 2026-10-30 04:09:58.194611899 +0000 UTC m=+5278.843660318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-secret" (UniqueName: "kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-tls-secret") pod "data-science-lokistack-gateway-686554c9b7-2s7ml" (UID: "4d802189-cb0b-41c2-ba62-e11231925023") : secret "data-science-lokistack-gateway-http" not found
Oct 30 04:09:57.695210 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.695079    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-lokistack-gateway\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.695996 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.695856    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/4d802189-cb0b-41c2-ba62-e11231925023-tenants\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.695996 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.695963    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.696091 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.696035    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.716222 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.716166    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpcvc\" (UniqueName: \"kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-kube-api-access-kpcvc\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:57.727433 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.727408    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"]
Oct 30 04:09:57.729052 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:09:57.729031    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e1f157_4854_45fa_81cf_454c7841587a.slice/crio-2f9c5cdf6c3c9b99b201a4781f552026543566a49648cdb7422273ce87ada262 WatchSource:0}: Error finding container 2f9c5cdf6c3c9b99b201a4781f552026543566a49648cdb7422273ce87ada262: Status 404 returned error can't find the container with id 2f9c5cdf6c3c9b99b201a4781f552026543566a49648cdb7422273ce87ada262
Oct 30 04:09:57.797865 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.797837    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:57.894592 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:57.894564    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"]
Oct 30 04:09:57.899886 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:09:57.898748    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89348102_649b_4362_892e_51db15bb9e18.slice/crio-4e4c7b4a5d08a94d5a493ad9cf1409bd8e23ac9d4a1f629130f87115a1a395ab WatchSource:0}: Error finding container 4e4c7b4a5d08a94d5a493ad9cf1409bd8e23ac9d4a1f629130f87115a1a395ab: Status 404 returned error can't find the container with id 4e4c7b4a5d08a94d5a493ad9cf1409bd8e23ac9d4a1f629130f87115a1a395ab
Oct 30 04:09:58.197202 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:58.197138    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-tls-secret\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:58.199812 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:58.199788    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-tls-secret\") pod \"data-science-lokistack-gateway-686554c9b7-2s7ml\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:58.447898 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:58.447822    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:58.549746 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:58.549718    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9" event={"ID":"89348102-649b-4362-892e-51db15bb9e18","Type":"ContainerStarted","Data":"30e25f29a5c593def9c436db8389713bcaf13fe352577dbdf7b2b718ed289737"}
Oct 30 04:09:58.549916 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:58.549766    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9" event={"ID":"89348102-649b-4362-892e-51db15bb9e18","Type":"ContainerStarted","Data":"4e4c7b4a5d08a94d5a493ad9cf1409bd8e23ac9d4a1f629130f87115a1a395ab"}
Oct 30 04:09:58.549916 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:58.549837    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:09:58.551505 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:58.551480    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx" event={"ID":"37e1f157-4854-45fa-81cf-454c7841587a","Type":"ContainerStarted","Data":"6c5aed6245cec119f16afef6fa1376db42fd73859aa3064ef0762107ce391433"}
Oct 30 04:09:58.551630 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:58.551524    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx" event={"ID":"37e1f157-4854-45fa-81cf-454c7841587a","Type":"ContainerStarted","Data":"2f9c5cdf6c3c9b99b201a4781f552026543566a49648cdb7422273ce87ada262"}
Oct 30 04:09:58.551630 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:58.551557    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:09:58.552935 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:58.552914    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr" event={"ID":"4f6549b0-c354-45d7-a9ba-ec2680ec4372","Type":"ContainerStarted","Data":"b458db8e2b3c737f3be578d2216e6a8eadd7e4613a79e4e4ae96761092514d14"}
Oct 30 04:09:58.553034 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:58.552948    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr" event={"ID":"4f6549b0-c354-45d7-a9ba-ec2680ec4372","Type":"ContainerStarted","Data":"550e29d58cdf9bfb313fb85947e3f8c1796c304b8b1786fe7f5aaf5c4241bd1a"}
Oct 30 04:09:58.553084 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:58.553045    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:09:58.592411 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:58.592373    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9" podStartSLOduration=1.592362181 podStartE2EDuration="1.592362181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:09:58.592098301 +0000 UTC m=+5279.241146723" watchObservedRunningTime="2026-10-30 04:09:58.592362181 +0000 UTC m=+5279.241410602"
Oct 30 04:09:58.633630 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:58.633594    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr" podStartSLOduration=2.633581922 podStartE2EDuration="2.633581922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:09:58.632695506 +0000 UTC m=+5279.281743927" watchObservedRunningTime="2026-10-30 04:09:58.633581922 +0000 UTC m=+5279.282630345"
Oct 30 04:09:58.676431 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:58.676394    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx" podStartSLOduration=1.676381944 podStartE2EDuration="1.676381944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:09:58.675740465 +0000 UTC m=+5279.324788885" watchObservedRunningTime="2026-10-30 04:09:58.676381944 +0000 UTC m=+5279.325430365"
Oct 30 04:09:58.750670 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:58.750618    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"]
Oct 30 04:09:58.752867 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:09:58.752843    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d802189_cb0b_41c2_ba62_e11231925023.slice/crio-90675980a1c07d99804448b2e15f1dc80fffa4f64adcf3da46a1d6ec24887f09 WatchSource:0}: Error finding container 90675980a1c07d99804448b2e15f1dc80fffa4f64adcf3da46a1d6ec24887f09: Status 404 returned error can't find the container with id 90675980a1c07d99804448b2e15f1dc80fffa4f64adcf3da46a1d6ec24887f09
Oct 30 04:09:59.558891 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:59.558855    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml" event={"ID":"4d802189-cb0b-41c2-ba62-e11231925023","Type":"ContainerStarted","Data":"a20bb6b5b1402f5d8c5ba6877b713a9d3cee4bbfc57caf461b48a0b172091cf0"}
Oct 30 04:09:59.559477 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:59.559452    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:59.559603 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:59.559495    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:59.559603 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:59.559516    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml" event={"ID":"4d802189-cb0b-41c2-ba62-e11231925023","Type":"ContainerStarted","Data":"fed01691ac5689e37c18067b636d78863fddda6ccd70c712deb8a19fef16fa88"}
Oct 30 04:09:59.559603 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:59.559541    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml" event={"ID":"4d802189-cb0b-41c2-ba62-e11231925023","Type":"ContainerStarted","Data":"90675980a1c07d99804448b2e15f1dc80fffa4f64adcf3da46a1d6ec24887f09"}
Oct 30 04:09:59.565101 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:59.565080    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:59.568307 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:59.568284    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:09:59.595404 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:59.595354    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:59.603897 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:09:59.603858    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml" podStartSLOduration=2.603844774 podStartE2EDuration="2.603844774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:09:59.602443834 +0000 UTC m=+5280.251492279" watchObservedRunningTime="2026-10-30 04:09:59.603844774 +0000 UTC m=+5280.252893221"
Oct 30 04:10:19.565828 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:10:19.565790    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:10:19.566315 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:10:19.566160    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:10:19.566315 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:10:19.566295    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:10:23.954688 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:10:23.954652    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:10:53.955224 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:10:53.955162    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:10:59.728927 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:10:59.728895    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:11:23.953298 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:11:23.953264    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:11:42.105593 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.105555    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"]
Oct 30 04:11:42.109144 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.109122    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:42.140125 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.140099    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-grpc\""
Oct 30 04:11:42.140272 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.140155    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-http\""
Oct 30 04:11:42.141333 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.141310    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"]
Oct 30 04:11:42.264324 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.264299    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:42.264420 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.264351    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:42.264478 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.264460    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1df57552-952a-4bfc-836e-75fc6bc9d5e3-config\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:42.264516 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.264486    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:42.264516 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.264503    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:42.264586 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.264560    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxwmp\" (UniqueName: \"kubernetes.io/projected/1df57552-952a-4bfc-836e-75fc6bc9d5e3-kube-api-access-wxwmp\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:42.365472 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.365414    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:42.365562 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.365477    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1df57552-952a-4bfc-836e-75fc6bc9d5e3-config\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:42.365562 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.365498    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:42.365562 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.365519    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:42.365562 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.365543    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxwmp\" (UniqueName: \"kubernetes.io/projected/1df57552-952a-4bfc-836e-75fc6bc9d5e3-kube-api-access-wxwmp\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:42.365736 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.365690    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:42.366191 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.366163    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:42.366570 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.366438    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1df57552-952a-4bfc-836e-75fc6bc9d5e3-config\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:42.368230 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.368200    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:42.368561 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.368535    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:42.368715 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.368700    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:42.386123 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:42.386099    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxwmp\" (UniqueName: \"kubernetes.io/projected/1df57552-952a-4bfc-836e-75fc6bc9d5e3-kube-api-access-wxwmp\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:47.502396 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:47.502358    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-936a7126-5683-41c9-899c-dcab125a2108\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ae481f104a8cef7d\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:47.502845 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:47.502431    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-354b3a14-d107-4b0d-a319-d0e427ea30eb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bbde2f5ca1f07e8b\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:47.533982 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:47.533948    4960 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-936a7126-5683-41c9-899c-dcab125a2108\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ae481f104a8cef7d\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:47.534112 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:47.533990    4960 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-354b3a14-d107-4b0d-a319-d0e427ea30eb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bbde2f5ca1f07e8b\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:47.603415 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:47.603391    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-354b3a14-d107-4b0d-a319-d0e427ea30eb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bbde2f5ca1f07e8b\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:47.603539 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:47.603455    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-936a7126-5683-41c9-899c-dcab125a2108\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ae481f104a8cef7d\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:47.603539 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:47.603519    4960 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-354b3a14-d107-4b0d-a319-d0e427ea30eb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bbde2f5ca1f07e8b\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:47.603611 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:47.603547    4960 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-936a7126-5683-41c9-899c-dcab125a2108\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ae481f104a8cef7d\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:47.619170 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:47.619141    4960 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-936a7126-5683-41c9-899c-dcab125a2108\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ae481f104a8cef7d\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") DevicePath \"csi-2db4e43b4fad1aab6b2031790fbd37302ad56657c5ba83765a2fdb82195c5786\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:47.619403 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:47.619219    4960 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-354b3a14-d107-4b0d-a319-d0e427ea30eb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bbde2f5ca1f07e8b\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") DevicePath \"csi-417401e30dcaa1c8c2b3f2af54e84992591373d59693fca364b52273eac7dd5a\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:47.725282 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:47.725253    4960 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-354b3a14-d107-4b0d-a319-d0e427ea30eb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bbde2f5ca1f07e8b\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/bd527562f7b97df8c633d0e271fb2c9c67e742337f9544fe3ba802e713564b9f/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:47.732803 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:47.732774    4960 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-936a7126-5683-41c9-899c-dcab125a2108\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ae481f104a8cef7d\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4e264e3e69579800e1690d3dbbd40e4e6742aab7b33d3263cc34dacdfcbc972d/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:47.758355 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:47.758303    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-354b3a14-d107-4b0d-a319-d0e427ea30eb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bbde2f5ca1f07e8b\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:47.759678 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:47.759656    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-936a7126-5683-41c9-899c-dcab125a2108\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ae481f104a8cef7d\") pod \"data-science-lokistack-ingester-2\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:47.821292 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:47.821264    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:48.124337 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:48.124310    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"]
Oct 30 04:11:48.125797 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:11:48.125772    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df57552_952a_4bfc_836e_75fc6bc9d5e3.slice/crio-e379a8dc3c40673943bca6814097e012844354e9fab14e0bb0e67a5123e74126 WatchSource:0}: Error finding container e379a8dc3c40673943bca6814097e012844354e9fab14e0bb0e67a5123e74126: Status 404 returned error can't find the container with id e379a8dc3c40673943bca6814097e012844354e9fab14e0bb0e67a5123e74126
Oct 30 04:11:49.046625 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:49.046589    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" event={"ID":"1df57552-952a-4bfc-836e-75fc6bc9d5e3","Type":"ContainerStarted","Data":"66aa35b968101725414d78df1ef8e55660ce13e3e4d4a181a875d2fb060d579e"}
Oct 30 04:11:49.047025 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:49.046648    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" event={"ID":"1df57552-952a-4bfc-836e-75fc6bc9d5e3","Type":"ContainerStarted","Data":"e379a8dc3c40673943bca6814097e012844354e9fab14e0bb0e67a5123e74126"}
Oct 30 04:11:49.047025 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:49.046705    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:11:49.090250 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:49.090203    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" podStartSLOduration=7.090171671 podStartE2EDuration="7.090171671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:11:49.089052277 +0000 UTC m=+5389.738100692" watchObservedRunningTime="2026-10-30 04:11:49.090171671 +0000 UTC m=+5389.739220091"
Oct 30 04:11:53.955380 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:11:53.955345    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:11:56.696006 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:56.695962    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:11:56.798153 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:56.798121    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:11:59.830125 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:11:59.830092    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:00.043344 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:00.043277    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 04:12:00.049492 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:00.049472    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 04:12:00.052261 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:00.052240    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 04:12:00.058266 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:00.058250    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 04:12:10.056912 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:10.056858    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" podUID="1df57552-952a-4bfc-836e-75fc6bc9d5e3" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:12:20.055049 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:20.055012    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" podUID="1df57552-952a-4bfc-836e-75fc6bc9d5e3" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:12:23.954075 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:12:23.954042    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:12:30.055981 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:30.055952    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:12:34.719978 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:34.719931    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:34.832417 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:34.832360    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:34.939441 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:34.939404    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:38.247114 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.247075    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:38.320893 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.320859    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"]
Oct 30 04:12:38.321148 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.321100    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" podUID="1df57552-952a-4bfc-836e-75fc6bc9d5e3" containerName="loki-ingester" containerID="cri-o://66aa35b968101725414d78df1ef8e55660ce13e3e4d4a181a875d2fb060d579e" gracePeriod=600
Oct 30 04:12:38.333518 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.333477    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"]
Oct 30 04:12:38.333800 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.333744    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr" podUID="4f6549b0-c354-45d7-a9ba-ec2680ec4372" containerName="loki-distributor" containerID="cri-o://b458db8e2b3c737f3be578d2216e6a8eadd7e4613a79e4e4ae96761092514d14" gracePeriod=30
Oct 30 04:12:38.338924 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.338891    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"]
Oct 30 04:12:38.340514 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.340478    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx" podUID="37e1f157-4854-45fa-81cf-454c7841587a" containerName="loki-querier" containerID="cri-o://6c5aed6245cec119f16afef6fa1376db42fd73859aa3064ef0762107ce391433" gracePeriod=30
Oct 30 04:12:38.341471 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.341299    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"]
Oct 30 04:12:38.341564 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.341507    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9" podUID="89348102-649b-4362-892e-51db15bb9e18" containerName="loki-query-frontend" containerID="cri-o://30e25f29a5c593def9c436db8389713bcaf13fe352577dbdf7b2b718ed289737" gracePeriod=30
Oct 30 04:12:38.348294 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.348258    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"]
Oct 30 04:12:38.348656 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.348629    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml" podUID="4d802189-cb0b-41c2-ba62-e11231925023" containerName="gateway" containerID="cri-o://fed01691ac5689e37c18067b636d78863fddda6ccd70c712deb8a19fef16fa88" gracePeriod=30
Oct 30 04:12:38.348956 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.348927    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml" podUID="4d802189-cb0b-41c2-ba62-e11231925023" containerName="opa" containerID="cri-o://a20bb6b5b1402f5d8c5ba6877b713a9d3cee4bbfc57caf461b48a0b172091cf0" gracePeriod=30
Oct 30 04:12:38.386797 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.386735    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:38.555758 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.555660    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:38.653442 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.653420    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:12:38.679611 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.679583    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:38.794605 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.794575    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89348102-649b-4362-892e-51db15bb9e18-kube-api-access-wvn6x\" (UniqueName: \"kubernetes.io/projected/89348102-649b-4362-892e-51db15bb9e18-kube-api-access-wvn6x\") pod \"89348102-649b-4362-892e-51db15bb9e18\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") "
Oct 30 04:12:38.794761 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.794640    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-query-frontend-http\") pod \"89348102-649b-4362-892e-51db15bb9e18\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") "
Oct 30 04:12:38.794761 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.794686    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-query-frontend-grpc\") pod \"89348102-649b-4362-892e-51db15bb9e18\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") "
Oct 30 04:12:38.794761 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.794712    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-ca-bundle\") pod \"89348102-649b-4362-892e-51db15bb9e18\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") "
Oct 30 04:12:38.794761 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.794756    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89348102-649b-4362-892e-51db15bb9e18-config\" (UniqueName: \"kubernetes.io/configmap/89348102-649b-4362-892e-51db15bb9e18-config\") pod \"89348102-649b-4362-892e-51db15bb9e18\" (UID: \"89348102-649b-4362-892e-51db15bb9e18\") "
Oct 30 04:12:38.795280 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.795146    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-ca-bundle" pod "89348102-649b-4362-892e-51db15bb9e18" (UID: "89348102-649b-4362-892e-51db15bb9e18"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:38.795401 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.795380    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89348102-649b-4362-892e-51db15bb9e18-config" pod "89348102-649b-4362-892e-51db15bb9e18" (UID: "89348102-649b-4362-892e-51db15bb9e18"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:38.797666 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.797638    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-query-frontend-grpc" pod "89348102-649b-4362-892e-51db15bb9e18" (UID: "89348102-649b-4362-892e-51db15bb9e18"). InnerVolumeSpecName "data-science-lokistack-query-frontend-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:38.797795 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.797756    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89348102-649b-4362-892e-51db15bb9e18-kube-api-access-wvn6x" pod "89348102-649b-4362-892e-51db15bb9e18" (UID: "89348102-649b-4362-892e-51db15bb9e18"). InnerVolumeSpecName "kube-api-access-wvn6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:38.798475 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.798453    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-query-frontend-http" pod "89348102-649b-4362-892e-51db15bb9e18" (UID: "89348102-649b-4362-892e-51db15bb9e18"). InnerVolumeSpecName "data-science-lokistack-query-frontend-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:38.895664 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.895621    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-query-frontend-http\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:38.895664 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.895654    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-query-frontend-grpc\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:38.895664 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.895669    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89348102-649b-4362-892e-51db15bb9e18-data-science-lokistack-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:38.895878 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.895680    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/89348102-649b-4362-892e-51db15bb9e18-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:38.895878 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.895691    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvn6x\" (UniqueName: \"kubernetes.io/projected/89348102-649b-4362-892e-51db15bb9e18-kube-api-access-wvn6x\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:38.918776 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:38.918755    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:12:39.097083 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.097003    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37e1f157-4854-45fa-81cf-454c7841587a-config\" (UniqueName: \"kubernetes.io/configmap/37e1f157-4854-45fa-81cf-454c7841587a-config\") pod \"37e1f157-4854-45fa-81cf-454c7841587a\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") "
Oct 30 04:12:39.097083 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.097075    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37e1f157-4854-45fa-81cf-454c7841587a-kube-api-access-w2b2s\" (UniqueName: \"kubernetes.io/projected/37e1f157-4854-45fa-81cf-454c7841587a-kube-api-access-w2b2s\") pod \"37e1f157-4854-45fa-81cf-454c7841587a\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") "
Oct 30 04:12:39.097355 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.097133    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-querier-http\") pod \"37e1f157-4854-45fa-81cf-454c7841587a\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") "
Oct 30 04:12:39.097355 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.097164    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-querier-grpc\") pod \"37e1f157-4854-45fa-81cf-454c7841587a\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") "
Oct 30 04:12:39.097355 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.097239    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-ca-bundle\") pod \"37e1f157-4854-45fa-81cf-454c7841587a\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") "
Oct 30 04:12:39.097355 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.097275    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-test-loki-lifecycle-secret\") pod \"37e1f157-4854-45fa-81cf-454c7841587a\" (UID: \"37e1f157-4854-45fa-81cf-454c7841587a\") "
Oct 30 04:12:39.097585 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.097499    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37e1f157-4854-45fa-81cf-454c7841587a-config" pod "37e1f157-4854-45fa-81cf-454c7841587a" (UID: "37e1f157-4854-45fa-81cf-454c7841587a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:39.097835 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.097810    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-ca-bundle" pod "37e1f157-4854-45fa-81cf-454c7841587a" (UID: "37e1f157-4854-45fa-81cf-454c7841587a"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:39.099829 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.099793    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e1f157-4854-45fa-81cf-454c7841587a-kube-api-access-w2b2s" pod "37e1f157-4854-45fa-81cf-454c7841587a" (UID: "37e1f157-4854-45fa-81cf-454c7841587a"). InnerVolumeSpecName "kube-api-access-w2b2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:39.099945 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.099865    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-test-loki-lifecycle-secret" pod "37e1f157-4854-45fa-81cf-454c7841587a" (UID: "37e1f157-4854-45fa-81cf-454c7841587a"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:39.100049 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.100025    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-querier-http" pod "37e1f157-4854-45fa-81cf-454c7841587a" (UID: "37e1f157-4854-45fa-81cf-454c7841587a"). InnerVolumeSpecName "data-science-lokistack-querier-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:39.101035 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.101014    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-querier-grpc" pod "37e1f157-4854-45fa-81cf-454c7841587a" (UID: "37e1f157-4854-45fa-81cf-454c7841587a"). InnerVolumeSpecName "data-science-lokistack-querier-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:39.195506 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.195481    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:12:39.198551 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.198525    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2b2s\" (UniqueName: \"kubernetes.io/projected/37e1f157-4854-45fa-81cf-454c7841587a-kube-api-access-w2b2s\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.198674 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.198552    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-querier-http\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.198674 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.198568    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-querier-grpc\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.198674 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.198582    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e1f157-4854-45fa-81cf-454c7841587a-data-science-lokistack-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.198674 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.198595    4960 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/37e1f157-4854-45fa-81cf-454c7841587a-test-loki-lifecycle-secret\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.198674 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.198610    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37e1f157-4854-45fa-81cf-454c7841587a-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.263550 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.263528    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:12:39.283914 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.283887    4960 generic.go:356] "Generic (PLEG): container finished" podID="4d802189-cb0b-41c2-ba62-e11231925023" containerID="a20bb6b5b1402f5d8c5ba6877b713a9d3cee4bbfc57caf461b48a0b172091cf0" exitCode=0
Oct 30 04:12:39.284039 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.283965    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml" event={"ID":"4d802189-cb0b-41c2-ba62-e11231925023","Type":"ContainerDied","Data":"a20bb6b5b1402f5d8c5ba6877b713a9d3cee4bbfc57caf461b48a0b172091cf0"}
Oct 30 04:12:39.285145 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.285122    4960 generic.go:356] "Generic (PLEG): container finished" podID="37e1f157-4854-45fa-81cf-454c7841587a" containerID="6c5aed6245cec119f16afef6fa1376db42fd73859aa3064ef0762107ce391433" exitCode=0
Oct 30 04:12:39.285274 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.285169    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"
Oct 30 04:12:39.285274 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.285212    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx" event={"ID":"37e1f157-4854-45fa-81cf-454c7841587a","Type":"ContainerDied","Data":"6c5aed6245cec119f16afef6fa1376db42fd73859aa3064ef0762107ce391433"}
Oct 30 04:12:39.285274 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.285256    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx" event={"ID":"37e1f157-4854-45fa-81cf-454c7841587a","Type":"ContainerDied","Data":"2f9c5cdf6c3c9b99b201a4781f552026543566a49648cdb7422273ce87ada262"}
Oct 30 04:12:39.285274 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.285276    4960 scope.go:122] "RemoveContainer" containerID="6c5aed6245cec119f16afef6fa1376db42fd73859aa3064ef0762107ce391433"
Oct 30 04:12:39.288434 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.287693    4960 generic.go:356] "Generic (PLEG): container finished" podID="4f6549b0-c354-45d7-a9ba-ec2680ec4372" containerID="b458db8e2b3c737f3be578d2216e6a8eadd7e4613a79e4e4ae96761092514d14" exitCode=0
Oct 30 04:12:39.288434 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.287796    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr" event={"ID":"4f6549b0-c354-45d7-a9ba-ec2680ec4372","Type":"ContainerDied","Data":"b458db8e2b3c737f3be578d2216e6a8eadd7e4613a79e4e4ae96761092514d14"}
Oct 30 04:12:39.288434 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.287829    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr" event={"ID":"4f6549b0-c354-45d7-a9ba-ec2680ec4372","Type":"ContainerDied","Data":"550e29d58cdf9bfb313fb85947e3f8c1796c304b8b1786fe7f5aaf5c4241bd1a"}
Oct 30 04:12:39.288434 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.287884    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"
Oct 30 04:12:39.291310 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.291287    4960 generic.go:356] "Generic (PLEG): container finished" podID="1df57552-952a-4bfc-836e-75fc6bc9d5e3" containerID="66aa35b968101725414d78df1ef8e55660ce13e3e4d4a181a875d2fb060d579e" exitCode=0
Oct 30 04:12:39.291405 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.291333    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:12:39.291405 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.291365    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" event={"ID":"1df57552-952a-4bfc-836e-75fc6bc9d5e3","Type":"ContainerDied","Data":"66aa35b968101725414d78df1ef8e55660ce13e3e4d4a181a875d2fb060d579e"}
Oct 30 04:12:39.291529 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.291413    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" event={"ID":"1df57552-952a-4bfc-836e-75fc6bc9d5e3","Type":"ContainerDied","Data":"e379a8dc3c40673943bca6814097e012844354e9fab14e0bb0e67a5123e74126"}
Oct 30 04:12:39.292986 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.292962    4960 generic.go:356] "Generic (PLEG): container finished" podID="89348102-649b-4362-892e-51db15bb9e18" containerID="30e25f29a5c593def9c436db8389713bcaf13fe352577dbdf7b2b718ed289737" exitCode=0
Oct 30 04:12:39.293107 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.292999    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9" event={"ID":"89348102-649b-4362-892e-51db15bb9e18","Type":"ContainerDied","Data":"30e25f29a5c593def9c436db8389713bcaf13fe352577dbdf7b2b718ed289737"}
Oct 30 04:12:39.293107 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.293010    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"
Oct 30 04:12:39.293107 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.293025    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9" event={"ID":"89348102-649b-4362-892e-51db15bb9e18","Type":"ContainerDied","Data":"4e4c7b4a5d08a94d5a493ad9cf1409bd8e23ac9d4a1f629130f87115a1a395ab"}
Oct 30 04:12:39.298357 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.298337    4960 scope.go:122] "RemoveContainer" containerID="6c5aed6245cec119f16afef6fa1376db42fd73859aa3064ef0762107ce391433"
Oct 30 04:12:39.298604 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:12:39.298586    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5aed6245cec119f16afef6fa1376db42fd73859aa3064ef0762107ce391433\": container with ID starting with 6c5aed6245cec119f16afef6fa1376db42fd73859aa3064ef0762107ce391433 not found: ID does not exist" containerID="6c5aed6245cec119f16afef6fa1376db42fd73859aa3064ef0762107ce391433"
Oct 30 04:12:39.298665 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.298611    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5aed6245cec119f16afef6fa1376db42fd73859aa3064ef0762107ce391433"} err="failed to get container status \"6c5aed6245cec119f16afef6fa1376db42fd73859aa3064ef0762107ce391433\": rpc error: code = NotFound desc = could not find container \"6c5aed6245cec119f16afef6fa1376db42fd73859aa3064ef0762107ce391433\": container with ID starting with 6c5aed6245cec119f16afef6fa1376db42fd73859aa3064ef0762107ce391433 not found: ID does not exist"
Oct 30 04:12:39.298665 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.298627    4960 scope.go:122] "RemoveContainer" containerID="b458db8e2b3c737f3be578d2216e6a8eadd7e4613a79e4e4ae96761092514d14"
Oct 30 04:12:39.298847 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.298793    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f6549b0-c354-45d7-a9ba-ec2680ec4372-kube-api-access-9bpsb\" (UniqueName: \"kubernetes.io/projected/4f6549b0-c354-45d7-a9ba-ec2680ec4372-kube-api-access-9bpsb\") pod \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") "
Oct 30 04:12:39.298959 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.298847    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f6549b0-c354-45d7-a9ba-ec2680ec4372-config\" (UniqueName: \"kubernetes.io/configmap/4f6549b0-c354-45d7-a9ba-ec2680ec4372-config\") pod \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") "
Oct 30 04:12:39.298959 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.298886    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-ca-bundle\") pod \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") "
Oct 30 04:12:39.299104 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.298986    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-distributor-http\") pod \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") "
Oct 30 04:12:39.299104 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.299028    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-distributor-grpc\") pod \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\" (UID: \"4f6549b0-c354-45d7-a9ba-ec2680ec4372\") "
Oct 30 04:12:39.299678 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.299653    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6549b0-c354-45d7-a9ba-ec2680ec4372-config" pod "4f6549b0-c354-45d7-a9ba-ec2680ec4372" (UID: "4f6549b0-c354-45d7-a9ba-ec2680ec4372"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:39.299823 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.299736    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-ca-bundle" pod "4f6549b0-c354-45d7-a9ba-ec2680ec4372" (UID: "4f6549b0-c354-45d7-a9ba-ec2680ec4372"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:39.302408 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.302381    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6549b0-c354-45d7-a9ba-ec2680ec4372-kube-api-access-9bpsb" pod "4f6549b0-c354-45d7-a9ba-ec2680ec4372" (UID: "4f6549b0-c354-45d7-a9ba-ec2680ec4372"). InnerVolumeSpecName "kube-api-access-9bpsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:39.302660 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.302634    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-distributor-grpc" pod "4f6549b0-c354-45d7-a9ba-ec2680ec4372" (UID: "4f6549b0-c354-45d7-a9ba-ec2680ec4372"). InnerVolumeSpecName "data-science-lokistack-distributor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:39.303453 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.303432    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-distributor-http" pod "4f6549b0-c354-45d7-a9ba-ec2680ec4372" (UID: "4f6549b0-c354-45d7-a9ba-ec2680ec4372"). InnerVolumeSpecName "data-science-lokistack-distributor-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:39.312578 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.312561    4960 scope.go:122] "RemoveContainer" containerID="b458db8e2b3c737f3be578d2216e6a8eadd7e4613a79e4e4ae96761092514d14"
Oct 30 04:12:39.312887 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:12:39.312868    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b458db8e2b3c737f3be578d2216e6a8eadd7e4613a79e4e4ae96761092514d14\": container with ID starting with b458db8e2b3c737f3be578d2216e6a8eadd7e4613a79e4e4ae96761092514d14 not found: ID does not exist" containerID="b458db8e2b3c737f3be578d2216e6a8eadd7e4613a79e4e4ae96761092514d14"
Oct 30 04:12:39.312964 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.312897    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b458db8e2b3c737f3be578d2216e6a8eadd7e4613a79e4e4ae96761092514d14"} err="failed to get container status \"b458db8e2b3c737f3be578d2216e6a8eadd7e4613a79e4e4ae96761092514d14\": rpc error: code = NotFound desc = could not find container \"b458db8e2b3c737f3be578d2216e6a8eadd7e4613a79e4e4ae96761092514d14\": container with ID starting with b458db8e2b3c737f3be578d2216e6a8eadd7e4613a79e4e4ae96761092514d14 not found: ID does not exist"
Oct 30 04:12:39.312964 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.312919    4960 scope.go:122] "RemoveContainer" containerID="66aa35b968101725414d78df1ef8e55660ce13e3e4d4a181a875d2fb060d579e"
Oct 30 04:12:39.321272 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.321257    4960 scope.go:122] "RemoveContainer" containerID="66aa35b968101725414d78df1ef8e55660ce13e3e4d4a181a875d2fb060d579e"
Oct 30 04:12:39.321578 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:12:39.321554    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66aa35b968101725414d78df1ef8e55660ce13e3e4d4a181a875d2fb060d579e\": container with ID starting with 66aa35b968101725414d78df1ef8e55660ce13e3e4d4a181a875d2fb060d579e not found: ID does not exist" containerID="66aa35b968101725414d78df1ef8e55660ce13e3e4d4a181a875d2fb060d579e"
Oct 30 04:12:39.321675 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.321582    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66aa35b968101725414d78df1ef8e55660ce13e3e4d4a181a875d2fb060d579e"} err="failed to get container status \"66aa35b968101725414d78df1ef8e55660ce13e3e4d4a181a875d2fb060d579e\": rpc error: code = NotFound desc = could not find container \"66aa35b968101725414d78df1ef8e55660ce13e3e4d4a181a875d2fb060d579e\": container with ID starting with 66aa35b968101725414d78df1ef8e55660ce13e3e4d4a181a875d2fb060d579e not found: ID does not exist"
Oct 30 04:12:39.321675 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.321599    4960 scope.go:122] "RemoveContainer" containerID="30e25f29a5c593def9c436db8389713bcaf13fe352577dbdf7b2b718ed289737"
Oct 30 04:12:39.329443 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.329427    4960 scope.go:122] "RemoveContainer" containerID="30e25f29a5c593def9c436db8389713bcaf13fe352577dbdf7b2b718ed289737"
Oct 30 04:12:39.329707 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:12:39.329690    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e25f29a5c593def9c436db8389713bcaf13fe352577dbdf7b2b718ed289737\": container with ID starting with 30e25f29a5c593def9c436db8389713bcaf13fe352577dbdf7b2b718ed289737 not found: ID does not exist" containerID="30e25f29a5c593def9c436db8389713bcaf13fe352577dbdf7b2b718ed289737"
Oct 30 04:12:39.329773 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.329715    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e25f29a5c593def9c436db8389713bcaf13fe352577dbdf7b2b718ed289737"} err="failed to get container status \"30e25f29a5c593def9c436db8389713bcaf13fe352577dbdf7b2b718ed289737\": rpc error: code = NotFound desc = could not find container \"30e25f29a5c593def9c436db8389713bcaf13fe352577dbdf7b2b718ed289737\": container with ID starting with 30e25f29a5c593def9c436db8389713bcaf13fe352577dbdf7b2b718ed289737 not found: ID does not exist"
Oct 30 04:12:39.361868 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.361753    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"]
Oct 30 04:12:39.365721 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.365690    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-hnwjx"]
Oct 30 04:12:39.399727 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.399709    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bbde2f5ca1f07e8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bbde2f5ca1f07e8b\") pod \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") "
Oct 30 04:12:39.399817 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.399744    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1df57552-952a-4bfc-836e-75fc6bc9d5e3-kube-api-access-wxwmp\" (UniqueName: \"kubernetes.io/projected/1df57552-952a-4bfc-836e-75fc6bc9d5e3-kube-api-access-wxwmp\") pod \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") "
Oct 30 04:12:39.399817 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.399777    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-test-loki-lifecycle-secret\") pod \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") "
Oct 30 04:12:39.399901 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.399846    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ae481f104a8cef7d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ae481f104a8cef7d\") pod \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") "
Oct 30 04:12:39.399946 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.399910    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ca-bundle\") pod \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") "
Oct 30 04:12:39.399982 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.399966    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ingester-grpc\") pod \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") "
Oct 30 04:12:39.400034 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.399999    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ingester-http\") pod \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") "
Oct 30 04:12:39.400095 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.400044    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1df57552-952a-4bfc-836e-75fc6bc9d5e3-config\" (UniqueName: \"kubernetes.io/configmap/1df57552-952a-4bfc-836e-75fc6bc9d5e3-config\") pod \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\" (UID: \"1df57552-952a-4bfc-836e-75fc6bc9d5e3\") "
Oct 30 04:12:39.400333 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.400316    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bpsb\" (UniqueName: \"kubernetes.io/projected/4f6549b0-c354-45d7-a9ba-ec2680ec4372-kube-api-access-9bpsb\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.400427 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.400338    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f6549b0-c354-45d7-a9ba-ec2680ec4372-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.400427 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.400352    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.400427 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.400368    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-distributor-http\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.400427 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.400384    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/4f6549b0-c354-45d7-a9ba-ec2680ec4372-data-science-lokistack-distributor-grpc\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.400988 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.400866    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df57552-952a-4bfc-836e-75fc6bc9d5e3-config" pod "1df57552-952a-4bfc-836e-75fc6bc9d5e3" (UID: "1df57552-952a-4bfc-836e-75fc6bc9d5e3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:39.400988 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.400953    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ca-bundle" pod "1df57552-952a-4bfc-836e-75fc6bc9d5e3" (UID: "1df57552-952a-4bfc-836e-75fc6bc9d5e3"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:39.402493 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.402467    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ingester-http" pod "1df57552-952a-4bfc-836e-75fc6bc9d5e3" (UID: "1df57552-952a-4bfc-836e-75fc6bc9d5e3"). InnerVolumeSpecName "data-science-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:39.402600 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.402533    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df57552-952a-4bfc-836e-75fc6bc9d5e3-kube-api-access-wxwmp" pod "1df57552-952a-4bfc-836e-75fc6bc9d5e3" (UID: "1df57552-952a-4bfc-836e-75fc6bc9d5e3"). InnerVolumeSpecName "kube-api-access-wxwmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:39.402820 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.402796    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-test-loki-lifecycle-secret" pod "1df57552-952a-4bfc-836e-75fc6bc9d5e3" (UID: "1df57552-952a-4bfc-836e-75fc6bc9d5e3"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:39.403120 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.403072    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ingester-grpc" pod "1df57552-952a-4bfc-836e-75fc6bc9d5e3" (UID: "1df57552-952a-4bfc-836e-75fc6bc9d5e3"). InnerVolumeSpecName "data-science-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:39.403251 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.403236    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0bbde2f5ca1f07e8b" pod "1df57552-952a-4bfc-836e-75fc6bc9d5e3" (UID: "1df57552-952a-4bfc-836e-75fc6bc9d5e3"). InnerVolumeSpecName "pvc-354b3a14-d107-4b0d-a319-d0e427ea30eb". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:12:39.403312 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.403284    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ae481f104a8cef7d" pod "1df57552-952a-4bfc-836e-75fc6bc9d5e3" (UID: "1df57552-952a-4bfc-836e-75fc6bc9d5e3"). InnerVolumeSpecName "pvc-936a7126-5683-41c9-899c-dcab125a2108". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:12:39.417247 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.417222    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"]
Oct 30 04:12:39.421408 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.421385    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-xbkk9"]
Oct 30 04:12:39.501054 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.501022    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.501054 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.501051    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ingester-grpc\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.501054 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.501061    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-data-science-lokistack-ingester-http\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.501407 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.501071    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1df57552-952a-4bfc-836e-75fc6bc9d5e3-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.501407 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.501100    4960 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-354b3a14-d107-4b0d-a319-d0e427ea30eb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bbde2f5ca1f07e8b\") on node \"ip-10-0-128-241.ec2.internal\" "
Oct 30 04:12:39.501407 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.501110    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxwmp\" (UniqueName: \"kubernetes.io/projected/1df57552-952a-4bfc-836e-75fc6bc9d5e3-kube-api-access-wxwmp\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.501407 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.501121    4960 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/1df57552-952a-4bfc-836e-75fc6bc9d5e3-test-loki-lifecycle-secret\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.501407 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.501134    4960 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-936a7126-5683-41c9-899c-dcab125a2108\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ae481f104a8cef7d\") on node \"ip-10-0-128-241.ec2.internal\" "
Oct 30 04:12:39.506681 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.506653    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37e1f157-4854-45fa-81cf-454c7841587a" path="/var/lib/kubelet/pods/37e1f157-4854-45fa-81cf-454c7841587a/volumes"
Oct 30 04:12:39.507103 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.507085    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89348102-649b-4362-892e-51db15bb9e18" path="/var/lib/kubelet/pods/89348102-649b-4362-892e-51db15bb9e18/volumes"
Oct 30 04:12:39.516379 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.516359    4960 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-936a7126-5683-41c9-899c-dcab125a2108" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ae481f104a8cef7d") on node "ip-10-0-128-241.ec2.internal"
Oct 30 04:12:39.516912 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.516897    4960 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-354b3a14-d107-4b0d-a319-d0e427ea30eb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0bbde2f5ca1f07e8b") on node "ip-10-0-128-241.ec2.internal"
Oct 30 04:12:39.596454 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.596432    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:12:39.602295 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.602265    4960 reconciler_common.go:299] "Volume detached for volume \"pvc-354b3a14-d107-4b0d-a319-d0e427ea30eb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bbde2f5ca1f07e8b\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"csi-417401e30dcaa1c8c2b3f2af54e84992591373d59693fca364b52273eac7dd5a\""
Oct 30 04:12:39.602427 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.602365    4960 reconciler_common.go:299] "Volume detached for volume \"pvc-936a7126-5683-41c9-899c-dcab125a2108\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ae481f104a8cef7d\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"csi-2db4e43b4fad1aab6b2031790fbd37302ad56657c5ba83765a2fdb82195c5786\""
Oct 30 04:12:39.629145 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.629076    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"]
Oct 30 04:12:39.630901 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.630876    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jdtjr"]
Oct 30 04:12:39.702930 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.702891    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-tls-secret\" (UniqueName: \"kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-tls-secret\") pod \"4d802189-cb0b-41c2-ba62-e11231925023\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") "
Oct 30 04:12:39.703072 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.702945    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-kube-api-access-kpcvc\" (UniqueName: \"kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-kube-api-access-kpcvc\") pod \"4d802189-cb0b-41c2-ba62-e11231925023\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") "
Oct 30 04:12:39.703072 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.702983    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-gateway-ca-bundle\") pod \"4d802189-cb0b-41c2-ba62-e11231925023\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") "
Oct 30 04:12:39.703072 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.703024    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-rbac\" (UniqueName: \"kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-rbac\") pod \"4d802189-cb0b-41c2-ba62-e11231925023\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") "
Oct 30 04:12:39.703072 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.703062    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d802189-cb0b-41c2-ba62-e11231925023-tenants\" (UniqueName: \"kubernetes.io/secret/4d802189-cb0b-41c2-ba62-e11231925023-tenants\") pod \"4d802189-cb0b-41c2-ba62-e11231925023\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") "
Oct 30 04:12:39.703320 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.703121    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-lokistack-gateway\") pod \"4d802189-cb0b-41c2-ba62-e11231925023\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") "
Oct 30 04:12:39.703320 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.703195    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-gateway-client-http\") pod \"4d802189-cb0b-41c2-ba62-e11231925023\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") "
Oct 30 04:12:39.703320 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.703251    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-ca-bundle\") pod \"4d802189-cb0b-41c2-ba62-e11231925023\" (UID: \"4d802189-cb0b-41c2-ba62-e11231925023\") "
Oct 30 04:12:39.703883 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.703816    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-ca-bundle" pod "4d802189-cb0b-41c2-ba62-e11231925023" (UID: "4d802189-cb0b-41c2-ba62-e11231925023"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:39.705802 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.705776    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-kube-api-access-kpcvc" pod "4d802189-cb0b-41c2-ba62-e11231925023" (UID: "4d802189-cb0b-41c2-ba62-e11231925023"). InnerVolumeSpecName "kube-api-access-kpcvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:39.705802 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.705791    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-gateway-ca-bundle" pod "4d802189-cb0b-41c2-ba62-e11231925023" (UID: "4d802189-cb0b-41c2-ba62-e11231925023"). InnerVolumeSpecName "data-science-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:39.705927 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.705812    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-tls-secret" pod "4d802189-cb0b-41c2-ba62-e11231925023" (UID: "4d802189-cb0b-41c2-ba62-e11231925023"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:39.706362 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.706336    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-gateway-client-http" pod "4d802189-cb0b-41c2-ba62-e11231925023" (UID: "4d802189-cb0b-41c2-ba62-e11231925023"). InnerVolumeSpecName "data-science-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:39.708769 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.708743    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-lokistack-gateway" pod "4d802189-cb0b-41c2-ba62-e11231925023" (UID: "4d802189-cb0b-41c2-ba62-e11231925023"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:39.709457 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.709437    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-rbac" pod "4d802189-cb0b-41c2-ba62-e11231925023" (UID: "4d802189-cb0b-41c2-ba62-e11231925023"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:39.710880 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.710859    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d802189-cb0b-41c2-ba62-e11231925023-tenants" pod "4d802189-cb0b-41c2-ba62-e11231925023" (UID: "4d802189-cb0b-41c2-ba62-e11231925023"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:39.711660 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.711641    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"]
Oct 30 04:12:39.717441 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.717420    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"]
Oct 30 04:12:39.803916 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.803888    4960 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/4d802189-cb0b-41c2-ba62-e11231925023-tenants\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.803916 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.803913    4960 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-lokistack-gateway\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.804057 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.803923    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-gateway-client-http\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.804057 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.803935    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.804057 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.803945    4960 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-tls-secret\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.804057 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.803954    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpcvc\" (UniqueName: \"kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-kube-api-access-kpcvc\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.804057 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.803962    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/4d802189-cb0b-41c2-ba62-e11231925023-data-science-lokistack-gateway-ca-bundle\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.804057 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:39.803972    4960 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/4d802189-cb0b-41c2-ba62-e11231925023-rbac\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:12:40.299590 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:40.299559    4960 generic.go:356] "Generic (PLEG): container finished" podID="4d802189-cb0b-41c2-ba62-e11231925023" containerID="fed01691ac5689e37c18067b636d78863fddda6ccd70c712deb8a19fef16fa88" exitCode=0
Oct 30 04:12:40.300046 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:40.299606    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"
Oct 30 04:12:40.300046 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:40.299644    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml" event={"ID":"4d802189-cb0b-41c2-ba62-e11231925023","Type":"ContainerDied","Data":"fed01691ac5689e37c18067b636d78863fddda6ccd70c712deb8a19fef16fa88"}
Oct 30 04:12:40.300046 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:40.299699    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml" event={"ID":"4d802189-cb0b-41c2-ba62-e11231925023","Type":"ContainerDied","Data":"90675980a1c07d99804448b2e15f1dc80fffa4f64adcf3da46a1d6ec24887f09"}
Oct 30 04:12:40.300046 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:40.299720    4960 scope.go:122] "RemoveContainer" containerID="a20bb6b5b1402f5d8c5ba6877b713a9d3cee4bbfc57caf461b48a0b172091cf0"
Oct 30 04:12:40.309693 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:40.309677    4960 scope.go:122] "RemoveContainer" containerID="fed01691ac5689e37c18067b636d78863fddda6ccd70c712deb8a19fef16fa88"
Oct 30 04:12:40.318485 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:40.318468    4960 scope.go:122] "RemoveContainer" containerID="a20bb6b5b1402f5d8c5ba6877b713a9d3cee4bbfc57caf461b48a0b172091cf0"
Oct 30 04:12:40.318762 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:12:40.318744    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a20bb6b5b1402f5d8c5ba6877b713a9d3cee4bbfc57caf461b48a0b172091cf0\": container with ID starting with a20bb6b5b1402f5d8c5ba6877b713a9d3cee4bbfc57caf461b48a0b172091cf0 not found: ID does not exist" containerID="a20bb6b5b1402f5d8c5ba6877b713a9d3cee4bbfc57caf461b48a0b172091cf0"
Oct 30 04:12:40.318819 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:40.318769    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20bb6b5b1402f5d8c5ba6877b713a9d3cee4bbfc57caf461b48a0b172091cf0"} err="failed to get container status \"a20bb6b5b1402f5d8c5ba6877b713a9d3cee4bbfc57caf461b48a0b172091cf0\": rpc error: code = NotFound desc = could not find container \"a20bb6b5b1402f5d8c5ba6877b713a9d3cee4bbfc57caf461b48a0b172091cf0\": container with ID starting with a20bb6b5b1402f5d8c5ba6877b713a9d3cee4bbfc57caf461b48a0b172091cf0 not found: ID does not exist"
Oct 30 04:12:40.318819 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:40.318785    4960 scope.go:122] "RemoveContainer" containerID="fed01691ac5689e37c18067b636d78863fddda6ccd70c712deb8a19fef16fa88"
Oct 30 04:12:40.319044 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:12:40.319026    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fed01691ac5689e37c18067b636d78863fddda6ccd70c712deb8a19fef16fa88\": container with ID starting with fed01691ac5689e37c18067b636d78863fddda6ccd70c712deb8a19fef16fa88 not found: ID does not exist" containerID="fed01691ac5689e37c18067b636d78863fddda6ccd70c712deb8a19fef16fa88"
Oct 30 04:12:40.319086 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:40.319052    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fed01691ac5689e37c18067b636d78863fddda6ccd70c712deb8a19fef16fa88"} err="failed to get container status \"fed01691ac5689e37c18067b636d78863fddda6ccd70c712deb8a19fef16fa88\": rpc error: code = NotFound desc = could not find container \"fed01691ac5689e37c18067b636d78863fddda6ccd70c712deb8a19fef16fa88\": container with ID starting with fed01691ac5689e37c18067b636d78863fddda6ccd70c712deb8a19fef16fa88 not found: ID does not exist"
Oct 30 04:12:40.354487 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:40.354466    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"]
Oct 30 04:12:40.368210 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:40.368173    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml"]
Oct 30 04:12:40.559749 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:40.559680    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml" podUID="4d802189-cb0b-41c2-ba62-e11231925023" containerName="opa" probeResult="failure" output="Get \"https://10.132.0.110:8083/ready\": context deadline exceeded"
Oct 30 04:12:40.559749 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:40.559706    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-2s7ml" podUID="4d802189-cb0b-41c2-ba62-e11231925023" containerName="gateway" probeResult="failure" output="Get \"https://10.132.0.110:8081/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:12:41.503119 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:41.503078    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1df57552-952a-4bfc-836e-75fc6bc9d5e3" path="/var/lib/kubelet/pods/1df57552-952a-4bfc-836e-75fc6bc9d5e3/volumes"
Oct 30 04:12:41.503659 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:41.503636    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d802189-cb0b-41c2-ba62-e11231925023" path="/var/lib/kubelet/pods/4d802189-cb0b-41c2-ba62-e11231925023/volumes"
Oct 30 04:12:41.504042 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:41.504028    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f6549b0-c354-45d7-a9ba-ec2680ec4372" path="/var/lib/kubelet/pods/4f6549b0-c354-45d7-a9ba-ec2680ec4372/volumes"
Oct 30 04:12:53.953264 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:12:53.953230    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:12:59.964418 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:12:59.964338    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:07.656098 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:07.656063    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:10.086854 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.086820    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:10.193501 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.193471    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:10.304528 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.304496    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:10.366224 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.366159    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"]
Oct 30 04:13:10.366506 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.366492    4960 cpu_manager.go:414] "containerMap: removing container" podUID="4d802189-cb0b-41c2-ba62-e11231925023" containerName="gateway"
Oct 30 04:13:10.366548 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.366510    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d802189-cb0b-41c2-ba62-e11231925023" containerName="gateway"
Oct 30 04:13:10.366548 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.366538    4960 cpu_manager.go:414] "containerMap: removing container" podUID="37e1f157-4854-45fa-81cf-454c7841587a" containerName="loki-querier"
Oct 30 04:13:10.366548 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.366544    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37e1f157-4854-45fa-81cf-454c7841587a" containerName="loki-querier"
Oct 30 04:13:10.366644 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.366552    4960 cpu_manager.go:414] "containerMap: removing container" podUID="4d802189-cb0b-41c2-ba62-e11231925023" containerName="opa"
Oct 30 04:13:10.366644 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.366557    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d802189-cb0b-41c2-ba62-e11231925023" containerName="opa"
Oct 30 04:13:10.366644 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.366567    4960 cpu_manager.go:414] "containerMap: removing container" podUID="89348102-649b-4362-892e-51db15bb9e18" containerName="loki-query-frontend"
Oct 30 04:13:10.366644 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.366572    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89348102-649b-4362-892e-51db15bb9e18" containerName="loki-query-frontend"
Oct 30 04:13:10.366644 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.366591    4960 cpu_manager.go:414] "containerMap: removing container" podUID="4f6549b0-c354-45d7-a9ba-ec2680ec4372" containerName="loki-distributor"
Oct 30 04:13:10.366644 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.366600    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f6549b0-c354-45d7-a9ba-ec2680ec4372" containerName="loki-distributor"
Oct 30 04:13:10.366644 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.366607    4960 cpu_manager.go:414] "containerMap: removing container" podUID="1df57552-952a-4bfc-836e-75fc6bc9d5e3" containerName="loki-ingester"
Oct 30 04:13:10.366644 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.366612    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df57552-952a-4bfc-836e-75fc6bc9d5e3" containerName="loki-ingester"
Oct 30 04:13:10.366898 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.366660    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="37e1f157-4854-45fa-81cf-454c7841587a" containerName="loki-querier"
Oct 30 04:13:10.366898 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.366670    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d802189-cb0b-41c2-ba62-e11231925023" containerName="opa"
Oct 30 04:13:10.366898 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.366681    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="89348102-649b-4362-892e-51db15bb9e18" containerName="loki-query-frontend"
Oct 30 04:13:10.366898 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.366688    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f6549b0-c354-45d7-a9ba-ec2680ec4372" containerName="loki-distributor"
Oct 30 04:13:10.366898 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.366694    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df57552-952a-4bfc-836e-75fc6bc9d5e3" containerName="loki-ingester"
Oct 30 04:13:10.366898 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.366699    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d802189-cb0b-41c2-ba62-e11231925023" containerName="gateway"
Oct 30 04:13:10.371296 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.371274    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:10.390897 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.390876    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:13:10.401267 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.401063    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:13:10.401267 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.401119    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 04:13:10.401267 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.401216    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:13:10.401510 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.401296    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\""
Oct 30 04:13:10.402444 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.402421    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"]
Oct 30 04:13:10.404486 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.404466    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-hjxcz\""
Oct 30 04:13:10.404583 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.404546    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\""
Oct 30 04:13:10.417172 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.417152    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k8nhl\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:10.417279 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.417255    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtxbj\" (UniqueName: \"kubernetes.io/projected/2d5782f1-d219-41cf-addd-375dd8c3cb1d-kube-api-access-gtxbj\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k8nhl\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:10.417325 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.417283    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2d5782f1-d219-41cf-addd-375dd8c3cb1d-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k8nhl\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:10.417373 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.417350    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k8nhl\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:10.417373 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.417369    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k8nhl\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:10.430689 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.430667    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:10.484009 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.483986    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"]
Oct 30 04:13:10.487607 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.487590    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"
Oct 30 04:13:10.505970 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.505943    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"]
Oct 30 04:13:10.518514 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.518489    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/19b85040-196e-468c-939d-8b93b291d593-tls\") pod \"korrel8r-5cbcc89b8f-ppfph\" (UID: \"19b85040-196e-468c-939d-8b93b291d593\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"
Oct 30 04:13:10.518598 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.518525    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k8nhl\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:10.518598 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.518554    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19b85040-196e-468c-939d-8b93b291d593-config\") pod \"korrel8r-5cbcc89b8f-ppfph\" (UID: \"19b85040-196e-468c-939d-8b93b291d593\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"
Oct 30 04:13:10.518677 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.518620    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49sfq\" (UniqueName: \"kubernetes.io/projected/19b85040-196e-468c-939d-8b93b291d593-kube-api-access-49sfq\") pod \"korrel8r-5cbcc89b8f-ppfph\" (UID: \"19b85040-196e-468c-939d-8b93b291d593\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"
Oct 30 04:13:10.518677 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.518673    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtxbj\" (UniqueName: \"kubernetes.io/projected/2d5782f1-d219-41cf-addd-375dd8c3cb1d-kube-api-access-gtxbj\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k8nhl\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:10.518762 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.518694    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2d5782f1-d219-41cf-addd-375dd8c3cb1d-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k8nhl\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:10.518762 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.518714    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k8nhl\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:10.518762 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.518738    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k8nhl\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:10.518974 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:13:10.518884    4960 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found
Oct 30 04:13:10.518974 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:13:10.518940    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-prometheus-client-cert podName:2d5782f1-d219-41cf-addd-375dd8c3cb1d nodeName:}" failed. No retries permitted until 2026-10-30 04:13:11.018922961 +0000 UTC m=+5471.667971367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-k8nhl" (UID: "2d5782f1-d219-41cf-addd-375dd8c3cb1d") : secret "prometheus-operated-tls" not found
Oct 30 04:13:10.519426 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.519405    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2d5782f1-d219-41cf-addd-375dd8c3cb1d-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k8nhl\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:10.521348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.521328    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k8nhl\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:10.521482 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.521460    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k8nhl\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:10.523895 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.523879    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\""
Oct 30 04:13:10.524009 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.523991    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-9xps5\""
Oct 30 04:13:10.539467 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.539438    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtxbj\" (UniqueName: \"kubernetes.io/projected/2d5782f1-d219-41cf-addd-375dd8c3cb1d-kube-api-access-gtxbj\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k8nhl\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:10.545827 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.545805    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:10.591913 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.591888    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\""
Oct 30 04:13:10.619376 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.619327    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/19b85040-196e-468c-939d-8b93b291d593-tls\") pod \"korrel8r-5cbcc89b8f-ppfph\" (UID: \"19b85040-196e-468c-939d-8b93b291d593\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"
Oct 30 04:13:10.619452 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.619379    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19b85040-196e-468c-939d-8b93b291d593-config\") pod \"korrel8r-5cbcc89b8f-ppfph\" (UID: \"19b85040-196e-468c-939d-8b93b291d593\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"
Oct 30 04:13:10.619452 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.619405    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49sfq\" (UniqueName: \"kubernetes.io/projected/19b85040-196e-468c-939d-8b93b291d593-kube-api-access-49sfq\") pod \"korrel8r-5cbcc89b8f-ppfph\" (UID: \"19b85040-196e-468c-939d-8b93b291d593\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"
Oct 30 04:13:10.620026 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.620007    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19b85040-196e-468c-939d-8b93b291d593-config\") pod \"korrel8r-5cbcc89b8f-ppfph\" (UID: \"19b85040-196e-468c-939d-8b93b291d593\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"
Oct 30 04:13:10.621557 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.621534    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/19b85040-196e-468c-939d-8b93b291d593-tls\") pod \"korrel8r-5cbcc89b8f-ppfph\" (UID: \"19b85040-196e-468c-939d-8b93b291d593\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"
Oct 30 04:13:10.639382 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.639364    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49sfq\" (UniqueName: \"kubernetes.io/projected/19b85040-196e-468c-939d-8b93b291d593-kube-api-access-49sfq\") pod \"korrel8r-5cbcc89b8f-ppfph\" (UID: \"19b85040-196e-468c-939d-8b93b291d593\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"
Oct 30 04:13:10.796823 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.796799    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:10.799674 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:10.799655    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"
Oct 30 04:13:11.022165 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.022091    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k8nhl\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:11.024665 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.024638    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k8nhl\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:11.025008 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.024992    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:11.089945 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.089910    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"]
Oct 30 04:13:11.091737 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:13:11.091709    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b85040_196e_468c_939d_8b93b291d593.slice/crio-1553dce3818c6b959a43ab8bc8b9835221a3888ca8caa5f91963b374aedbd5f7 WatchSource:0}: Error finding container 1553dce3818c6b959a43ab8bc8b9835221a3888ca8caa5f91963b374aedbd5f7: Status 404 returned error can't find the container with id 1553dce3818c6b959a43ab8bc8b9835221a3888ca8caa5f91963b374aedbd5f7
Oct 30 04:13:11.201305 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.201274    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:11.284372 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.284305    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:11.385920 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.385860    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:11.442131 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.442097    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph" event={"ID":"19b85040-196e-468c-939d-8b93b291d593","Type":"ContainerStarted","Data":"a415dd5e4cad1ea9e56dd64918c8be62410d786f8ed0f788d34e6ad25f8d7f2f"}
Oct 30 04:13:11.442311 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.442148    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph" event={"ID":"19b85040-196e-468c-939d-8b93b291d593","Type":"ContainerStarted","Data":"1553dce3818c6b959a43ab8bc8b9835221a3888ca8caa5f91963b374aedbd5f7"}
Oct 30 04:13:11.442384 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.442324    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"
Oct 30 04:13:11.485145 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.485098    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph" podStartSLOduration=1.485085753 podStartE2EDuration="1.485085753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:11.483936741 +0000 UTC m=+5472.132985162" watchObservedRunningTime="2026-10-30 04:13:11.485085753 +0000 UTC m=+5472.134134192"
Oct 30 04:13:11.575736 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.575714    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"]
Oct 30 04:13:11.577609 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:13:11.577584    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5782f1_d219_41cf_addd_375dd8c3cb1d.slice/crio-f5255c4d0e86b8e2af03fb94ef8097755d30a9f3b0668796a306dca73d759f46 WatchSource:0}: Error finding container f5255c4d0e86b8e2af03fb94ef8097755d30a9f3b0668796a306dca73d759f46: Status 404 returned error can't find the container with id f5255c4d0e86b8e2af03fb94ef8097755d30a9f3b0668796a306dca73d759f46
Oct 30 04:13:11.882344 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.882266    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:13:11.887123 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.887098    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:11.906284 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.905608    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:13:11.906284 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.905650    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:13:11.909215 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.907275    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:13:11.918730 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.918696    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:13:11.918864 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.918801    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:13:11.918915 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.918871    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:13:11.929334 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.929305    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvs8x\" (UniqueName: \"kubernetes.io/projected/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-kube-api-access-vvs8x\") pod \"data-science-collector-collector-0\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:11.929445 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.929374    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:11.929445 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.929428    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:11.929560 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.929487    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:11.929613 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.929562    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:11.929690 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.929640    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:11.961215 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:11.961133    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:12.030876 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.030839    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvs8x\" (UniqueName: \"kubernetes.io/projected/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-kube-api-access-vvs8x\") pod \"data-science-collector-collector-0\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:12.031044 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.030900    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:12.031044 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.030924    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:12.031044 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.030956    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:12.031044 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.030983    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:12.031044 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.031036    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:12.031744 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.031710    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:12.034335 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.034308    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:12.034498 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.034447    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:12.034889 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.034841    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:12.035616 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.035595    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:12.060631 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.060595    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvs8x\" (UniqueName: \"kubernetes.io/projected/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-kube-api-access-vvs8x\") pod \"data-science-collector-collector-0\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:12.077150 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.077117    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:12.188218 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.188112    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:12.200008 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.199991    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:12.448403 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.448333    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl" event={"ID":"2d5782f1-d219-41cf-addd-375dd8c3cb1d","Type":"ContainerStarted","Data":"f8fb7c664610763ac203ff923eac5ac73a78cd2235daa43677adad820916f0c1"}
Oct 30 04:13:12.448526 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.448406    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl" event={"ID":"2d5782f1-d219-41cf-addd-375dd8c3cb1d","Type":"ContainerStarted","Data":"f5255c4d0e86b8e2af03fb94ef8097755d30a9f3b0668796a306dca73d759f46"}
Oct 30 04:13:12.488915 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.488855    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl" podStartSLOduration=2.488839247 podStartE2EDuration="2.488839247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:12.488145529 +0000 UTC m=+5473.137193949" watchObservedRunningTime="2026-10-30 04:13:12.488839247 +0000 UTC m=+5473.137887669"
Oct 30 04:13:12.499058 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.499037    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:13:12.501098 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:13:12.501079    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd675dd9_79e0_4ef9_9d72_3bf275c77ae3.slice/crio-44d1d52925c1ee65e0923d40263799840d4202a5ec1ba2913a395e508fad1a48 WatchSource:0}: Error finding container 44d1d52925c1ee65e0923d40263799840d4202a5ec1ba2913a395e508fad1a48: Status 404 returned error can't find the container with id 44d1d52925c1ee65e0923d40263799840d4202a5ec1ba2913a395e508fad1a48
Oct 30 04:13:12.503249 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.503219    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:13:12.503350 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:12.503282    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:13:13.454485 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:13.454452    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3","Type":"ContainerStarted","Data":"43f785dead7b43f08e15678a4077687b09cbdb3f9dbb928fa2bf74a22ce7a4fb"}
Oct 30 04:13:13.454858 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:13.454508    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3","Type":"ContainerStarted","Data":"44d1d52925c1ee65e0923d40263799840d4202a5ec1ba2913a395e508fad1a48"}
Oct 30 04:13:13.454858 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:13.454603    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:13.497245 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:13.497195    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.497167719 podStartE2EDuration="2.497167719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:13.4953679 +0000 UTC m=+5474.144416323" watchObservedRunningTime="2026-10-30 04:13:13.497167719 +0000 UTC m=+5474.146216139"
Oct 30 04:13:14.634319 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:14.634287    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:14.868545 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:14.868510    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:14.970016 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:14.969944    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:15.175786 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.175755    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:13:15.179446 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.179423    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.199742 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.199719    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\""
Oct 30 04:13:15.199896 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.199717    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:13:15.199896 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.199736    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\""
Oct 30 04:13:15.200098 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.199768    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\""
Oct 30 04:13:15.200098 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.199800    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-8ncjk\""
Oct 30 04:13:15.205545 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.205501    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:15.214693 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.214672    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:13:15.256641 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.256583    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf88a46a-5476-4ef1-8266-a42af9af6c59-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.256641 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.256621    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.256802 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.256642    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/bf88a46a-5476-4ef1-8266-a42af9af6c59-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.256802 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.256767    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.256901 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.256825    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.256901 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.256844    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf88a46a-5476-4ef1-8266-a42af9af6c59-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.256901 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.256865    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtchw\" (UniqueName: \"kubernetes.io/projected/bf88a46a-5476-4ef1-8266-a42af9af6c59-kube-api-access-xtchw\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.357449 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.357419    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.357603 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.357462    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.357603 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.357480    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf88a46a-5476-4ef1-8266-a42af9af6c59-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.357603 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.357497    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtchw\" (UniqueName: \"kubernetes.io/projected/bf88a46a-5476-4ef1-8266-a42af9af6c59-kube-api-access-xtchw\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.357603 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.357556    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf88a46a-5476-4ef1-8266-a42af9af6c59-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.357815 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.357767    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.357876 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.357810    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/bf88a46a-5476-4ef1-8266-a42af9af6c59-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.358406 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.358288    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/bf88a46a-5476-4ef1-8266-a42af9af6c59-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.360465 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.360440    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.360645 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.360616    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf88a46a-5476-4ef1-8266-a42af9af6c59-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.360759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.360698    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.360759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.360746    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf88a46a-5476-4ef1-8266-a42af9af6c59-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.361781 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.361755    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.394994 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.394960    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:15.395454 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.395431    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:13:15.396778 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.396751    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtchw\" (UniqueName: \"kubernetes.io/projected/bf88a46a-5476-4ef1-8266-a42af9af6c59-kube-api-access-xtchw\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.399702 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.399681    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.412510 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.412485    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:13:15.419453 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.419427    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 04:13:15.419594 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.419576    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 04:13:15.419665 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.419580    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-fw4nx\""
Oct 30 04:13:15.419665 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.419621    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 04:13:15.419665 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.419640    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 04:13:15.419665 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.419661    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:13:15.419898 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.419643    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 04:13:15.419898 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.419681    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:13:15.419898 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.419736    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 04:13:15.495584 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.495552    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:15.559935 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.559900    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78597359-922e-49db-b7e1-5fab26dbb467-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.560037 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.559953    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.560037 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.559994    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.560037 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.560024    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.560144 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.560059    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.560144 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.560108    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.560235 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.560156    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.560235 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.560210    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.560297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.560268    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.560334 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.560308    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78597359-922e-49db-b7e1-5fab26dbb467-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.560368 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.560338    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.560404 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.560363    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrrnh\" (UniqueName: \"kubernetes.io/projected/78597359-922e-49db-b7e1-5fab26dbb467-kube-api-access-hrrnh\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.661093 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.661062    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.661555 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.661102    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78597359-922e-49db-b7e1-5fab26dbb467-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.661555 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.661123    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.661555 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.661236    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrrnh\" (UniqueName: \"kubernetes.io/projected/78597359-922e-49db-b7e1-5fab26dbb467-kube-api-access-hrrnh\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.661555 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.661294    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78597359-922e-49db-b7e1-5fab26dbb467-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.661555 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.661325    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.661555 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.661362    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.661555 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.661409    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.661555 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.661506    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.662000 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.661580    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.662000 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.661634    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.662000 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.661670    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.662000 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.661785    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.662845 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.662801    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.663353 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.663328    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.664547 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.664527    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.664645 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.664557    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78597359-922e-49db-b7e1-5fab26dbb467-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.665305 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.665280    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78597359-922e-49db-b7e1-5fab26dbb467-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.665412 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.665288    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.665580 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.665555    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.666305 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.666285    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.666433 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.666410    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.666687 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.666671    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.683846 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.683819    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrrnh\" (UniqueName: \"kubernetes.io/projected/78597359-922e-49db-b7e1-5fab26dbb467-kube-api-access-hrrnh\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:15.801476 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:15.801373    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:13:15.803135 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:13:15.803111    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf88a46a_5476_4ef1_8266_a42af9af6c59.slice/crio-583cfaad5cf7d58cb8481aa7014fb3d56ff9ac401a4b43297c23cbc85236d220 WatchSource:0}: Error finding container 583cfaad5cf7d58cb8481aa7014fb3d56ff9ac401a4b43297c23cbc85236d220: Status 404 returned error can't find the container with id 583cfaad5cf7d58cb8481aa7014fb3d56ff9ac401a4b43297c23cbc85236d220
Oct 30 04:13:16.479366 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:16.479289    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"bf88a46a-5476-4ef1-8266-a42af9af6c59","Type":"ContainerStarted","Data":"cdaade17437bfffe3fbbdfbfa667a39cf5419e5b96c60d3bee08e36209bf2a9f"}
Oct 30 04:13:16.479366 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:16.479360    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"bf88a46a-5476-4ef1-8266-a42af9af6c59","Type":"ContainerStarted","Data":"583cfaad5cf7d58cb8481aa7014fb3d56ff9ac401a4b43297c23cbc85236d220"}
Oct 30 04:13:19.293273 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:19.293237    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:19.323881 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:19.323854    4960 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:19.394421 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:19.394395    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:19.394575 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:19.394544    4960 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:19.410256 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:19.410232    4960 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") DevicePath \"csi-6f8c8e4c1326657251c30594036033694cadf25e97408e4f9d9da436f78f3fa4\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:19.462125 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:19.462101    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:13:19.514701 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:19.514673    4960 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/8c516923ea0339579ebf808cef81b9e0aaf55e5b619e2c6187025fa3b2b51f7f/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:19.542917 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:19.542888    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:19.626113 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:19.626085    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:19.943749 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:19.943727    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:13:19.944144 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:13:19.944120    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78597359_922e_49db_b7e1_5fab26dbb467.slice/crio-58183ff5860c57a0f275419124d55bd4d34162c0554cf2fbf11b0529744157f3 WatchSource:0}: Error finding container 58183ff5860c57a0f275419124d55bd4d34162c0554cf2fbf11b0529744157f3: Status 404 returned error can't find the container with id 58183ff5860c57a0f275419124d55bd4d34162c0554cf2fbf11b0529744157f3
Oct 30 04:13:20.499798 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:20.499768    4960 generic.go:356] "Generic (PLEG): container finished" podID="bf88a46a-5476-4ef1-8266-a42af9af6c59" containerID="cdaade17437bfffe3fbbdfbfa667a39cf5419e5b96c60d3bee08e36209bf2a9f" exitCode=0
Oct 30 04:13:20.500205 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:20.499844    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"bf88a46a-5476-4ef1-8266-a42af9af6c59","Type":"ContainerDied","Data":"cdaade17437bfffe3fbbdfbfa667a39cf5419e5b96c60d3bee08e36209bf2a9f"}
Oct 30 04:13:20.501621 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:20.501599    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"78597359-922e-49db-b7e1-5fab26dbb467","Type":"ContainerStarted","Data":"39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43"}
Oct 30 04:13:20.501739 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:20.501659    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"78597359-922e-49db-b7e1-5fab26dbb467","Type":"ContainerStarted","Data":"58183ff5860c57a0f275419124d55bd4d34162c0554cf2fbf11b0529744157f3"}
Oct 30 04:13:21.507989 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:21.507954    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"bf88a46a-5476-4ef1-8266-a42af9af6c59","Type":"ContainerStarted","Data":"90de8c5bc87574cd83039d70bfd5256e0c1424d783ecc528029bb81b478e2d23"}
Oct 30 04:13:21.508390 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:21.508018    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"bf88a46a-5476-4ef1-8266-a42af9af6c59","Type":"ContainerStarted","Data":"9c27e3910d90b3c03d6db0c9ccbfeb841dc67132f0f8e02649495a4f9160c219"}
Oct 30 04:13:21.508390 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:21.508367    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:21.561015 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:21.560966    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.275015898 podStartE2EDuration="6.560953825s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.285937927s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:21.553748349 +0000 UTC m=+5482.202796771" watchObservedRunningTime="2026-10-30 04:13:21.560953825 +0000 UTC m=+5482.210002246"
Oct 30 04:13:22.450867 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:22.450839    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"
Oct 30 04:13:23.953323 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:13:23.953295    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:13:24.524075 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:24.524041    4960 generic.go:356] "Generic (PLEG): container finished" podID="78597359-922e-49db-b7e1-5fab26dbb467" containerID="39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43" exitCode=0
Oct 30 04:13:24.524243 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:24.524115    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"78597359-922e-49db-b7e1-5fab26dbb467","Type":"ContainerDied","Data":"39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43"}
Oct 30 04:13:25.531095 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:25.531056    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"78597359-922e-49db-b7e1-5fab26dbb467","Type":"ContainerStarted","Data":"fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76"}
Oct 30 04:13:25.531594 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:25.531141    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"78597359-922e-49db-b7e1-5fab26dbb467","Type":"ContainerStarted","Data":"e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149"}
Oct 30 04:13:25.531594 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:25.531198    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"78597359-922e-49db-b7e1-5fab26dbb467","Type":"ContainerStarted","Data":"940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88"}
Oct 30 04:13:25.608580 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:25.608524    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=7.022812937 podStartE2EDuration="10.608509817s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.58569688s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:25.594453828 +0000 UTC m=+5486.243502260" watchObservedRunningTime="2026-10-30 04:13:25.608509817 +0000 UTC m=+5486.257558238"
Oct 30 04:13:27.514687 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:27.514659    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:13:29.626713 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:29.626676    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:34.626939 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:34.626903    4960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:34.632711 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:34.632687    4960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:35.581202 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:35.581151    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:13:48.543872 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:48.543836    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:48.743988 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:48.743937    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:48.852751 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:48.852728    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:50.424830 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:50.424798    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:50.538351 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:50.538316    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:50.638757 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:50.638722    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:53.472587 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:53.472555    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:13:53.472964 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:53.472781    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="cd675dd9-79e0-4ef9-9d72-3bf275c77ae3" containerName="otc-container" containerID="cri-o://43f785dead7b43f08e15678a4077687b09cbdb3f9dbb928fa2bf74a22ce7a4fb" gracePeriod=30
Oct 30 04:13:53.954015 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:13:53.953980    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:13:54.671258 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.671223    4960 generic.go:356] "Generic (PLEG): container finished" podID="cd675dd9-79e0-4ef9-9d72-3bf275c77ae3" containerID="43f785dead7b43f08e15678a4077687b09cbdb3f9dbb928fa2bf74a22ce7a4fb" exitCode=0
Oct 30 04:13:54.671598 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.671297    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3","Type":"ContainerDied","Data":"43f785dead7b43f08e15678a4077687b09cbdb3f9dbb928fa2bf74a22ce7a4fb"}
Oct 30 04:13:54.723746 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.723720    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:54.876270 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.876244    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-otc-internal\" (UniqueName: \"kubernetes.io/configmap/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-otc-internal\") pod \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") "
Oct 30 04:13:54.876449 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.876281    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-otlp-tls-certs\") pod \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") "
Oct 30 04:13:54.876449 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.876322    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-kube-api-access-vvs8x\" (UniqueName: \"kubernetes.io/projected/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-kube-api-access-vvs8x\") pod \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") "
Oct 30 04:13:54.876449 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.876341    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-tls-certs\") pod \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") "
Oct 30 04:13:54.876629 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.876448    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-data-science-collector-ta-client-cert\") pod \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") "
Oct 30 04:13:54.876629 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.876504    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-monitoring-tls-certs\") pod \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\" (UID: \"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3\") "
Oct 30 04:13:54.876775 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.876747    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-otc-internal" pod "cd675dd9-79e0-4ef9-9d72-3bf275c77ae3" (UID: "cd675dd9-79e0-4ef9-9d72-3bf275c77ae3"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:13:54.876863 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.876846    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:13:54.878740 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.878703    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-tls-certs" pod "cd675dd9-79e0-4ef9-9d72-3bf275c77ae3" (UID: "cd675dd9-79e0-4ef9-9d72-3bf275c77ae3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:54.878864 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.878737    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-otlp-tls-certs" pod "cd675dd9-79e0-4ef9-9d72-3bf275c77ae3" (UID: "cd675dd9-79e0-4ef9-9d72-3bf275c77ae3"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:54.878864 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.878820    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-monitoring-tls-certs" pod "cd675dd9-79e0-4ef9-9d72-3bf275c77ae3" (UID: "cd675dd9-79e0-4ef9-9d72-3bf275c77ae3"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:54.879144 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.879122    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-data-science-collector-ta-client-cert" pod "cd675dd9-79e0-4ef9-9d72-3bf275c77ae3" (UID: "cd675dd9-79e0-4ef9-9d72-3bf275c77ae3"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:54.879144 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.879138    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-kube-api-access-vvs8x" pod "cd675dd9-79e0-4ef9-9d72-3bf275c77ae3" (UID: "cd675dd9-79e0-4ef9-9d72-3bf275c77ae3"). InnerVolumeSpecName "kube-api-access-vvs8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:13:54.977802 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.977779    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:13:54.977802 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.977801    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvs8x\" (UniqueName: \"kubernetes.io/projected/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-kube-api-access-vvs8x\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:13:54.977944 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.977811    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:13:54.977944 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.977820    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:13:54.977944 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:54.977829    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:13:55.677016 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.676986    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:55.677016 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.677002    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cd675dd9-79e0-4ef9-9d72-3bf275c77ae3","Type":"ContainerDied","Data":"44d1d52925c1ee65e0923d40263799840d4202a5ec1ba2913a395e508fad1a48"}
Oct 30 04:13:55.677483 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.677065    4960 scope.go:122] "RemoveContainer" containerID="43f785dead7b43f08e15678a4077687b09cbdb3f9dbb928fa2bf74a22ce7a4fb"
Oct 30 04:13:55.730362 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.730338    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:13:55.744160 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.744138    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:13:55.774030 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.772214    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:13:55.774030 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.773043    4960 cpu_manager.go:414] "containerMap: removing container" podUID="cd675dd9-79e0-4ef9-9d72-3bf275c77ae3" containerName="otc-container"
Oct 30 04:13:55.774030 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.773062    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd675dd9-79e0-4ef9-9d72-3bf275c77ae3" containerName="otc-container"
Oct 30 04:13:55.774030 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.773288    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd675dd9-79e0-4ef9-9d72-3bf275c77ae3" containerName="otc-container"
Oct 30 04:13:55.779011 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.778983    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:55.792699 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.792672    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:13:55.795095 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.795074    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-txvtm\""
Oct 30 04:13:55.795342 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.795319    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:13:55.795445 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.795330    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:13:55.795445 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.795339    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:13:55.795445 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.795391    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:13:55.795551 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.795469    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:13:55.885766 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.885740    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:55.885892 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.885770    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:55.885892 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.885790    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:55.885892 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.885806    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phfhz\" (UniqueName: \"kubernetes.io/projected/eeb4f69a-b405-473a-88cc-f05b3c140e70-kube-api-access-phfhz\") pod \"data-science-collector-collector-0\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:55.886067 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.885984    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:55.886067 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.886029    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/eeb4f69a-b405-473a-88cc-f05b3c140e70-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:55.986916 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.986884    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:55.987118 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.986925    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:55.987118 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.986952    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:55.987118 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.986974    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phfhz\" (UniqueName: \"kubernetes.io/projected/eeb4f69a-b405-473a-88cc-f05b3c140e70-kube-api-access-phfhz\") pod \"data-science-collector-collector-0\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:55.987118 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.987027    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:55.987118 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.987063    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/eeb4f69a-b405-473a-88cc-f05b3c140e70-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:55.988083 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.988053    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/eeb4f69a-b405-473a-88cc-f05b3c140e70-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:55.989577 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.989546    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:55.989667 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.989588    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:55.989775 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.989755    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:55.989824 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:55.989809    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:56.007113 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:56.007089    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phfhz\" (UniqueName: \"kubernetes.io/projected/eeb4f69a-b405-473a-88cc-f05b3c140e70-kube-api-access-phfhz\") pod \"data-science-collector-collector-0\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:56.092216 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:56.092192    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:13:56.387082 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:56.387056    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:13:56.388455 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:13:56.388429    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb4f69a_b405_473a_88cc_f05b3c140e70.slice/crio-566fbf4ff58126b3568622884d4426e22dcec43f4d4d0a68252ca9ca356726d9 WatchSource:0}: Error finding container 566fbf4ff58126b3568622884d4426e22dcec43f4d4d0a68252ca9ca356726d9: Status 404 returned error can't find the container with id 566fbf4ff58126b3568622884d4426e22dcec43f4d4d0a68252ca9ca356726d9
Oct 30 04:13:56.390648 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:56.390617    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:13:56.390735 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:56.390675    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:13:56.561546 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:56.561505    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:56.666814 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:56.666741    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:56.684691 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:56.684659    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"eeb4f69a-b405-473a-88cc-f05b3c140e70","Type":"ContainerStarted","Data":"ad2bbefb0fcc43ff603bc5de100646f6e881c8eb425748d361329275d97d7af6"}
Oct 30 04:13:56.685082 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:56.684723    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"eeb4f69a-b405-473a-88cc-f05b3c140e70","Type":"ContainerStarted","Data":"566fbf4ff58126b3568622884d4426e22dcec43f4d4d0a68252ca9ca356726d9"}
Oct 30 04:13:56.726459 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:56.726414    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.726401992 podStartE2EDuration="1.726401992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:56.725791364 +0000 UTC m=+5517.374839788" watchObservedRunningTime="2026-10-30 04:13:56.726401992 +0000 UTC m=+5517.375450413"
Oct 30 04:13:56.779688 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:56.779659    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:56.888230 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:56.888173    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:57.503746 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:57.503717    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd675dd9-79e0-4ef9-9d72-3bf275c77ae3" path="/var/lib/kubelet/pods/cd675dd9-79e0-4ef9-9d72-3bf275c77ae3/volumes"
Oct 30 04:13:58.433356 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:58.433323    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:58.545242 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:13:58.545210    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:00.065962 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:00.065934    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:00.515550 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:00.515512    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:14:00.515776 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:00.515735    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="eeb4f69a-b405-473a-88cc-f05b3c140e70" containerName="otc-container" containerID="cri-o://ad2bbefb0fcc43ff603bc5de100646f6e881c8eb425748d361329275d97d7af6" gracePeriod=30
Oct 30 04:14:00.705923 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:00.705892    4960 generic.go:356] "Generic (PLEG): container finished" podID="eeb4f69a-b405-473a-88cc-f05b3c140e70" containerID="ad2bbefb0fcc43ff603bc5de100646f6e881c8eb425748d361329275d97d7af6" exitCode=0
Oct 30 04:14:00.706058 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:00.705959    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"eeb4f69a-b405-473a-88cc-f05b3c140e70","Type":"ContainerDied","Data":"ad2bbefb0fcc43ff603bc5de100646f6e881c8eb425748d361329275d97d7af6"}
Oct 30 04:14:00.769860 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:00.769837    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:00.927445 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:00.927366    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-data-science-collector-ta-client-cert\") pod \"eeb4f69a-b405-473a-88cc-f05b3c140e70\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") "
Oct 30 04:14:00.927615 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:00.927458    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeb4f69a-b405-473a-88cc-f05b3c140e70-kube-api-access-phfhz\" (UniqueName: \"kubernetes.io/projected/eeb4f69a-b405-473a-88cc-f05b3c140e70-kube-api-access-phfhz\") pod \"eeb4f69a-b405-473a-88cc-f05b3c140e70\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") "
Oct 30 04:14:00.927615 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:00.927480    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-otlp-tls-certs\") pod \"eeb4f69a-b405-473a-88cc-f05b3c140e70\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") "
Oct 30 04:14:00.927615 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:00.927523    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-tls-certs\") pod \"eeb4f69a-b405-473a-88cc-f05b3c140e70\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") "
Oct 30 04:14:00.927615 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:00.927541    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-monitoring-tls-certs\") pod \"eeb4f69a-b405-473a-88cc-f05b3c140e70\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") "
Oct 30 04:14:00.927615 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:00.927601    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeb4f69a-b405-473a-88cc-f05b3c140e70-otc-internal\" (UniqueName: \"kubernetes.io/configmap/eeb4f69a-b405-473a-88cc-f05b3c140e70-otc-internal\") pod \"eeb4f69a-b405-473a-88cc-f05b3c140e70\" (UID: \"eeb4f69a-b405-473a-88cc-f05b3c140e70\") "
Oct 30 04:14:00.928207 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:00.928089    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb4f69a-b405-473a-88cc-f05b3c140e70-otc-internal" pod "eeb4f69a-b405-473a-88cc-f05b3c140e70" (UID: "eeb4f69a-b405-473a-88cc-f05b3c140e70"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:14:00.930349 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:00.930326    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-data-science-collector-ta-client-cert" pod "eeb4f69a-b405-473a-88cc-f05b3c140e70" (UID: "eeb4f69a-b405-473a-88cc-f05b3c140e70"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:00.930587 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:00.930566    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-tls-certs" pod "eeb4f69a-b405-473a-88cc-f05b3c140e70" (UID: "eeb4f69a-b405-473a-88cc-f05b3c140e70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:00.930657 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:00.930576    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb4f69a-b405-473a-88cc-f05b3c140e70-kube-api-access-phfhz" pod "eeb4f69a-b405-473a-88cc-f05b3c140e70" (UID: "eeb4f69a-b405-473a-88cc-f05b3c140e70"). InnerVolumeSpecName "kube-api-access-phfhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:14:00.930657 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:00.930582    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-otlp-tls-certs" pod "eeb4f69a-b405-473a-88cc-f05b3c140e70" (UID: "eeb4f69a-b405-473a-88cc-f05b3c140e70"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:00.930657 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:00.930593    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-monitoring-tls-certs" pod "eeb4f69a-b405-473a-88cc-f05b3c140e70" (UID: "eeb4f69a-b405-473a-88cc-f05b3c140e70"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:01.028870 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.028848    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phfhz\" (UniqueName: \"kubernetes.io/projected/eeb4f69a-b405-473a-88cc-f05b3c140e70-kube-api-access-phfhz\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:01.028870 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.028869    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:01.029015 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.028880    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:01.029015 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.028889    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:01.029015 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.028908    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/eeb4f69a-b405-473a-88cc-f05b3c140e70-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:01.029015 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.028918    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/eeb4f69a-b405-473a-88cc-f05b3c140e70-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:01.712325 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.712297    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:01.712724 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.712292    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"eeb4f69a-b405-473a-88cc-f05b3c140e70","Type":"ContainerDied","Data":"566fbf4ff58126b3568622884d4426e22dcec43f4d4d0a68252ca9ca356726d9"}
Oct 30 04:14:01.712724 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.712440    4960 scope.go:122] "RemoveContainer" containerID="ad2bbefb0fcc43ff603bc5de100646f6e881c8eb425748d361329275d97d7af6"
Oct 30 04:14:01.765954 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.765889    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:14:01.779228 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.779201    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:14:01.806171 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.806147    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:14:01.806540 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.806527    4960 cpu_manager.go:414] "containerMap: removing container" podUID="eeb4f69a-b405-473a-88cc-f05b3c140e70" containerName="otc-container"
Oct 30 04:14:01.806584 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.806546    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb4f69a-b405-473a-88cc-f05b3c140e70" containerName="otc-container"
Oct 30 04:14:01.806626 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.806617    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb4f69a-b405-473a-88cc-f05b3c140e70" containerName="otc-container"
Oct 30 04:14:01.809715 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.809693    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:01.829615 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.829592    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:14:01.831972 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.831950    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:14:01.832083 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.832063    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:14:01.832154 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.832109    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:14:01.832154 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.832146    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:14:01.832294 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.832205    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:14:01.832488 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.832464    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-txvtm\""
Oct 30 04:14:01.935437 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.935405    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:01.935568 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.935444    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:01.935568 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.935471    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:01.935661 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.935598    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:01.935661 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.935628    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnqj6\" (UniqueName: \"kubernetes.io/projected/93a0812d-ed41-455f-8371-0409cad2b544-kube-api-access-tnqj6\") pod \"data-science-collector-collector-0\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:01.935768 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:01.935663    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/93a0812d-ed41-455f-8371-0409cad2b544-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:02.036675 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:02.036610    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/93a0812d-ed41-455f-8371-0409cad2b544-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:02.036675 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:02.036660    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:02.036869 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:02.036700    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:02.036869 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:02.036735    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:02.037007 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:02.036882    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:02.037007 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:02.036925    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnqj6\" (UniqueName: \"kubernetes.io/projected/93a0812d-ed41-455f-8371-0409cad2b544-kube-api-access-tnqj6\") pod \"data-science-collector-collector-0\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:02.037568 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:02.037543    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/93a0812d-ed41-455f-8371-0409cad2b544-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:02.039753 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:02.039728    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:02.039890 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:02.039876    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:02.039931 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:02.039879    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:02.039931 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:02.039897    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:02.057405 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:02.057383    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnqj6\" (UniqueName: \"kubernetes.io/projected/93a0812d-ed41-455f-8371-0409cad2b544-kube-api-access-tnqj6\") pod \"data-science-collector-collector-0\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:02.121571 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:02.121548    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:02.409944 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:02.409923    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:14:02.412013 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:14:02.411981    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a0812d_ed41_455f_8371_0409cad2b544.slice/crio-bca590c2d5080e1aef6ae2dbe11784b11b7f441a5c866a6a3a7f9daf4297b353 WatchSource:0}: Error finding container bca590c2d5080e1aef6ae2dbe11784b11b7f441a5c866a6a3a7f9daf4297b353: Status 404 returned error can't find the container with id bca590c2d5080e1aef6ae2dbe11784b11b7f441a5c866a6a3a7f9daf4297b353
Oct 30 04:14:02.414252 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:02.414224    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:14:02.414317 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:02.414280    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:14:02.718983 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:02.718897    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"93a0812d-ed41-455f-8371-0409cad2b544","Type":"ContainerStarted","Data":"38b3eadc37d9e6dda2bfdca81e1b641751feac5ee1a77432343a8f05f6bf0db9"}
Oct 30 04:14:02.718983 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:02.718949    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"93a0812d-ed41-455f-8371-0409cad2b544","Type":"ContainerStarted","Data":"bca590c2d5080e1aef6ae2dbe11784b11b7f441a5c866a6a3a7f9daf4297b353"}
Oct 30 04:14:02.761976 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:02.761921    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.761901624 podStartE2EDuration="1.761901624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:02.760627025 +0000 UTC m=+5523.409675446" watchObservedRunningTime="2026-10-30 04:14:02.761901624 +0000 UTC m=+5523.410950050"
Oct 30 04:14:03.503043 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:03.503008    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeb4f69a-b405-473a-88cc-f05b3c140e70" path="/var/lib/kubelet/pods/eeb4f69a-b405-473a-88cc-f05b3c140e70/volumes"
Oct 30 04:14:23.952221 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:14:23.952135    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:14:28.306072 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:28.306038    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:28.413162 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:28.413116    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:28.523227 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:28.523193    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:30.074858 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:30.074819    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:30.184491 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:30.184455    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:30.290145 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:30.290085    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:33.682243 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:33.682208    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:14:33.682742 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:33.682421    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="93a0812d-ed41-455f-8371-0409cad2b544" containerName="otc-container" containerID="cri-o://38b3eadc37d9e6dda2bfdca81e1b641751feac5ee1a77432343a8f05f6bf0db9" gracePeriod=30
Oct 30 04:14:34.275434 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.275397    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"]
Oct 30 04:14:34.275625 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.275579    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl" podUID="2d5782f1-d219-41cf-addd-375dd8c3cb1d" containerName="kube-rbac-proxy" containerID="cri-o://f8fb7c664610763ac203ff923eac5ac73a78cd2235daa43677adad820916f0c1" gracePeriod=30
Oct 30 04:14:34.328447 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.328423    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"]
Oct 30 04:14:34.328616 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.328596    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph" podUID="19b85040-196e-468c-939d-8b93b291d593" containerName="korrel8r" containerID="cri-o://a415dd5e4cad1ea9e56dd64918c8be62410d786f8ed0f788d34e6ad25f8d7f2f" gracePeriod=30
Oct 30 04:14:34.460671 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.460636    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl" podUID="2d5782f1-d219-41cf-addd-375dd8c3cb1d" containerName="kube-rbac-proxy" probeResult="failure" output="dial tcp 10.132.0.112:8443: connect: connection refused"
Oct 30 04:14:34.607710 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.607682    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:14:34.609926 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.609900    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"
Oct 30 04:14:34.686479 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.686445    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:34.696152 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.696123    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-kube-rbac-proxy-config\") pod \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") "
Oct 30 04:14:34.696326 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.696172    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-prometheus-client-cert\") pod \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") "
Oct 30 04:14:34.696326 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.696231    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b85040-196e-468c-939d-8b93b291d593-config\" (UniqueName: \"kubernetes.io/configmap/19b85040-196e-468c-939d-8b93b291d593-config\") pod \"19b85040-196e-468c-939d-8b93b291d593\" (UID: \"19b85040-196e-468c-939d-8b93b291d593\") "
Oct 30 04:14:34.696326 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.696267    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19b85040-196e-468c-939d-8b93b291d593-tls\" (UniqueName: \"kubernetes.io/secret/19b85040-196e-468c-939d-8b93b291d593-tls\") pod \"19b85040-196e-468c-939d-8b93b291d593\" (UID: \"19b85040-196e-468c-939d-8b93b291d593\") "
Oct 30 04:14:34.696326 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.696302    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b85040-196e-468c-939d-8b93b291d593-kube-api-access-49sfq\" (UniqueName: \"kubernetes.io/projected/19b85040-196e-468c-939d-8b93b291d593-kube-api-access-49sfq\") pod \"19b85040-196e-468c-939d-8b93b291d593\" (UID: \"19b85040-196e-468c-939d-8b93b291d593\") "
Oct 30 04:14:34.696514 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.696357    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-tls\" (UniqueName: \"kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-tls\") pod \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") "
Oct 30 04:14:34.696514 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.696417    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d5782f1-d219-41cf-addd-375dd8c3cb1d-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2d5782f1-d219-41cf-addd-375dd8c3cb1d-prometheus-ca\") pod \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") "
Oct 30 04:14:34.696514 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.696487    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d5782f1-d219-41cf-addd-375dd8c3cb1d-kube-api-access-gtxbj\" (UniqueName: \"kubernetes.io/projected/2d5782f1-d219-41cf-addd-375dd8c3cb1d-kube-api-access-gtxbj\") pod \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\" (UID: \"2d5782f1-d219-41cf-addd-375dd8c3cb1d\") "
Oct 30 04:14:34.699682 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.699618    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d5782f1-d219-41cf-addd-375dd8c3cb1d-prometheus-ca" pod "2d5782f1-d219-41cf-addd-375dd8c3cb1d" (UID: "2d5782f1-d219-41cf-addd-375dd8c3cb1d"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:14:34.700008 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.699975    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b85040-196e-468c-939d-8b93b291d593-config" pod "19b85040-196e-468c-939d-8b93b291d593" (UID: "19b85040-196e-468c-939d-8b93b291d593"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:14:34.700204 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.700137    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5782f1-d219-41cf-addd-375dd8c3cb1d-kube-api-access-gtxbj" pod "2d5782f1-d219-41cf-addd-375dd8c3cb1d" (UID: "2d5782f1-d219-41cf-addd-375dd8c3cb1d"). InnerVolumeSpecName "kube-api-access-gtxbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:14:34.700371 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.700350    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19b85040-196e-468c-939d-8b93b291d593-tls" pod "19b85040-196e-468c-939d-8b93b291d593" (UID: "19b85040-196e-468c-939d-8b93b291d593"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:34.700649 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.700622    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-tls" pod "2d5782f1-d219-41cf-addd-375dd8c3cb1d" (UID: "2d5782f1-d219-41cf-addd-375dd8c3cb1d"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:34.700759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.700646    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-kube-rbac-proxy-config" pod "2d5782f1-d219-41cf-addd-375dd8c3cb1d" (UID: "2d5782f1-d219-41cf-addd-375dd8c3cb1d"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:34.703990 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.703967    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b85040-196e-468c-939d-8b93b291d593-kube-api-access-49sfq" pod "19b85040-196e-468c-939d-8b93b291d593" (UID: "19b85040-196e-468c-939d-8b93b291d593"). InnerVolumeSpecName "kube-api-access-49sfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:14:34.705679 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.705656    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-prometheus-client-cert" pod "2d5782f1-d219-41cf-addd-375dd8c3cb1d" (UID: "2d5782f1-d219-41cf-addd-375dd8c3cb1d"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:34.759292 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.759251    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:14:34.759575 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.759545    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="78597359-922e-49db-b7e1-5fab26dbb467" containerName="prometheus" containerID="cri-o://940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88" gracePeriod=600
Oct 30 04:14:34.759721 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.759588    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="78597359-922e-49db-b7e1-5fab26dbb467" containerName="thanos-sidecar" containerID="cri-o://fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76" gracePeriod=600
Oct 30 04:14:34.759721 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.759599    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="78597359-922e-49db-b7e1-5fab26dbb467" containerName="config-reloader" containerID="cri-o://e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149" gracePeriod=600
Oct 30 04:14:34.771503 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.769753    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:14:34.771503 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.770059    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="bf88a46a-5476-4ef1-8266-a42af9af6c59" containerName="alertmanager" containerID="cri-o://9c27e3910d90b3c03d6db0c9ccbfeb841dc67132f0f8e02649495a4f9160c219" gracePeriod=120
Oct 30 04:14:34.771503 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.770166    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="bf88a46a-5476-4ef1-8266-a42af9af6c59" containerName="config-reloader" containerID="cri-o://90de8c5bc87574cd83039d70bfd5256e0c1424d783ecc528029bb81b478e2d23" gracePeriod=120
Oct 30 04:14:34.798255 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.798231    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtxbj\" (UniqueName: \"kubernetes.io/projected/2d5782f1-d219-41cf-addd-375dd8c3cb1d-kube-api-access-gtxbj\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:34.798255 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.798257    4960 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-kube-rbac-proxy-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:34.798395 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.798268    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-prometheus-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:34.798395 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.798283    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19b85040-196e-468c-939d-8b93b291d593-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:34.798395 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.798292    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/19b85040-196e-468c-939d-8b93b291d593-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:34.798395 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.798300    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49sfq\" (UniqueName: \"kubernetes.io/projected/19b85040-196e-468c-939d-8b93b291d593-kube-api-access-49sfq\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:34.798395 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.798308    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2d5782f1-d219-41cf-addd-375dd8c3cb1d-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:34.798395 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.798316    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2d5782f1-d219-41cf-addd-375dd8c3cb1d-prometheus-ca\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:34.817628 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.817600    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:34.871668 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.871643    4960 generic.go:356] "Generic (PLEG): container finished" podID="2d5782f1-d219-41cf-addd-375dd8c3cb1d" containerID="f8fb7c664610763ac203ff923eac5ac73a78cd2235daa43677adad820916f0c1" exitCode=0
Oct 30 04:14:34.871779 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.871689    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"
Oct 30 04:14:34.871779 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.871715    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl" event={"ID":"2d5782f1-d219-41cf-addd-375dd8c3cb1d","Type":"ContainerDied","Data":"f8fb7c664610763ac203ff923eac5ac73a78cd2235daa43677adad820916f0c1"}
Oct 30 04:14:34.871779 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.871762    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl" event={"ID":"2d5782f1-d219-41cf-addd-375dd8c3cb1d","Type":"ContainerDied","Data":"f5255c4d0e86b8e2af03fb94ef8097755d30a9f3b0668796a306dca73d759f46"}
Oct 30 04:14:34.871991 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.871787    4960 scope.go:122] "RemoveContainer" containerID="f8fb7c664610763ac203ff923eac5ac73a78cd2235daa43677adad820916f0c1"
Oct 30 04:14:34.873280 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.873255    4960 generic.go:356] "Generic (PLEG): container finished" podID="19b85040-196e-468c-939d-8b93b291d593" containerID="a415dd5e4cad1ea9e56dd64918c8be62410d786f8ed0f788d34e6ad25f8d7f2f" exitCode=0
Oct 30 04:14:34.873373 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.873294    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"
Oct 30 04:14:34.873373 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.873292    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph" event={"ID":"19b85040-196e-468c-939d-8b93b291d593","Type":"ContainerDied","Data":"a415dd5e4cad1ea9e56dd64918c8be62410d786f8ed0f788d34e6ad25f8d7f2f"}
Oct 30 04:14:34.873443 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.873390    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph" event={"ID":"19b85040-196e-468c-939d-8b93b291d593","Type":"ContainerDied","Data":"1553dce3818c6b959a43ab8bc8b9835221a3888ca8caa5f91963b374aedbd5f7"}
Oct 30 04:14:34.885026 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.885000    4960 scope.go:122] "RemoveContainer" containerID="f8fb7c664610763ac203ff923eac5ac73a78cd2235daa43677adad820916f0c1"
Oct 30 04:14:34.885313 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:14:34.885285    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8fb7c664610763ac203ff923eac5ac73a78cd2235daa43677adad820916f0c1\": container with ID starting with f8fb7c664610763ac203ff923eac5ac73a78cd2235daa43677adad820916f0c1 not found: ID does not exist" containerID="f8fb7c664610763ac203ff923eac5ac73a78cd2235daa43677adad820916f0c1"
Oct 30 04:14:34.885422 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.885324    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8fb7c664610763ac203ff923eac5ac73a78cd2235daa43677adad820916f0c1"} err="failed to get container status \"f8fb7c664610763ac203ff923eac5ac73a78cd2235daa43677adad820916f0c1\": rpc error: code = NotFound desc = could not find container \"f8fb7c664610763ac203ff923eac5ac73a78cd2235daa43677adad820916f0c1\": container with ID starting with f8fb7c664610763ac203ff923eac5ac73a78cd2235daa43677adad820916f0c1 not found: ID does not exist"
Oct 30 04:14:34.885422 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.885344    4960 scope.go:122] "RemoveContainer" containerID="a415dd5e4cad1ea9e56dd64918c8be62410d786f8ed0f788d34e6ad25f8d7f2f"
Oct 30 04:14:34.893221 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.893199    4960 scope.go:122] "RemoveContainer" containerID="a415dd5e4cad1ea9e56dd64918c8be62410d786f8ed0f788d34e6ad25f8d7f2f"
Oct 30 04:14:34.893529 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:14:34.893510    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a415dd5e4cad1ea9e56dd64918c8be62410d786f8ed0f788d34e6ad25f8d7f2f\": container with ID starting with a415dd5e4cad1ea9e56dd64918c8be62410d786f8ed0f788d34e6ad25f8d7f2f not found: ID does not exist" containerID="a415dd5e4cad1ea9e56dd64918c8be62410d786f8ed0f788d34e6ad25f8d7f2f"
Oct 30 04:14:34.893590 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.893537    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a415dd5e4cad1ea9e56dd64918c8be62410d786f8ed0f788d34e6ad25f8d7f2f"} err="failed to get container status \"a415dd5e4cad1ea9e56dd64918c8be62410d786f8ed0f788d34e6ad25f8d7f2f\": rpc error: code = NotFound desc = could not find container \"a415dd5e4cad1ea9e56dd64918c8be62410d786f8ed0f788d34e6ad25f8d7f2f\": container with ID starting with a415dd5e4cad1ea9e56dd64918c8be62410d786f8ed0f788d34e6ad25f8d7f2f not found: ID does not exist"
Oct 30 04:14:34.928760 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.928737    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"]
Oct 30 04:14:34.933918 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.933895    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k8nhl"]
Oct 30 04:14:34.934325 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.934286    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:34.968689 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.968668    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"]
Oct 30 04:14:34.972805 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:34.972787    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ppfph"]
Oct 30 04:14:35.120526 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.120503    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:35.122799 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.122766    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:35.201259 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.201231    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-otlp-tls-certs\") pod \"93a0812d-ed41-455f-8371-0409cad2b544\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") "
Oct 30 04:14:35.201448 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.201265    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a0812d-ed41-455f-8371-0409cad2b544-otc-internal\" (UniqueName: \"kubernetes.io/configmap/93a0812d-ed41-455f-8371-0409cad2b544-otc-internal\") pod \"93a0812d-ed41-455f-8371-0409cad2b544\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") "
Oct 30 04:14:35.201448 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.201288    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-monitoring-tls-certs\") pod \"93a0812d-ed41-455f-8371-0409cad2b544\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") "
Oct 30 04:14:35.201448 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.201334    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-tls-certs\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-tls-certs\") pod \"93a0812d-ed41-455f-8371-0409cad2b544\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") "
Oct 30 04:14:35.201448 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.201358    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-data-science-collector-ta-client-cert\") pod \"93a0812d-ed41-455f-8371-0409cad2b544\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") "
Oct 30 04:14:35.201448 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.201403    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a0812d-ed41-455f-8371-0409cad2b544-kube-api-access-tnqj6\" (UniqueName: \"kubernetes.io/projected/93a0812d-ed41-455f-8371-0409cad2b544-kube-api-access-tnqj6\") pod \"93a0812d-ed41-455f-8371-0409cad2b544\" (UID: \"93a0812d-ed41-455f-8371-0409cad2b544\") "
Oct 30 04:14:35.201722 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.201692    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a0812d-ed41-455f-8371-0409cad2b544-otc-internal" pod "93a0812d-ed41-455f-8371-0409cad2b544" (UID: "93a0812d-ed41-455f-8371-0409cad2b544"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:14:35.203559 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.203527    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-otlp-tls-certs" pod "93a0812d-ed41-455f-8371-0409cad2b544" (UID: "93a0812d-ed41-455f-8371-0409cad2b544"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:35.203694 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.203657    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-tls-certs" pod "93a0812d-ed41-455f-8371-0409cad2b544" (UID: "93a0812d-ed41-455f-8371-0409cad2b544"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:35.203745 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.203707    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-data-science-collector-ta-client-cert" pod "93a0812d-ed41-455f-8371-0409cad2b544" (UID: "93a0812d-ed41-455f-8371-0409cad2b544"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:35.204024 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.204000    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-monitoring-tls-certs" pod "93a0812d-ed41-455f-8371-0409cad2b544" (UID: "93a0812d-ed41-455f-8371-0409cad2b544"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:35.204024 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.204015    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a0812d-ed41-455f-8371-0409cad2b544-kube-api-access-tnqj6" pod "93a0812d-ed41-455f-8371-0409cad2b544" (UID: "93a0812d-ed41-455f-8371-0409cad2b544"). InnerVolumeSpecName "kube-api-access-tnqj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:14:35.302596 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.302548    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.302596 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.302569    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/93a0812d-ed41-455f-8371-0409cad2b544-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.302596 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.302578    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.302596 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.302587    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.302596 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.302597    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/93a0812d-ed41-455f-8371-0409cad2b544-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.302817 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.302606    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnqj6\" (UniqueName: \"kubernetes.io/projected/93a0812d-ed41-455f-8371-0409cad2b544-kube-api-access-tnqj6\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.512496 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.512452    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b85040-196e-468c-939d-8b93b291d593" path="/var/lib/kubelet/pods/19b85040-196e-468c-939d-8b93b291d593/volumes"
Oct 30 04:14:35.513203 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.513003    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d5782f1-d219-41cf-addd-375dd8c3cb1d" path="/var/lib/kubelet/pods/2d5782f1-d219-41cf-addd-375dd8c3cb1d/volumes"
Oct 30 04:14:35.521693 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.521675    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:35.606443 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.606417    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"78597359-922e-49db-b7e1-5fab26dbb467\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") "
Oct 30 04:14:35.606604 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.606465    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78597359-922e-49db-b7e1-5fab26dbb467-config-out\" (UniqueName: \"kubernetes.io/empty-dir/78597359-922e-49db-b7e1-5fab26dbb467-config-out\") pod \"78597359-922e-49db-b7e1-5fab26dbb467\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") "
Oct 30 04:14:35.606604 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.606515    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"78597359-922e-49db-b7e1-5fab26dbb467\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") "
Oct 30 04:14:35.606604 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.606547    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78597359-922e-49db-b7e1-5fab26dbb467-tls-assets\" (UniqueName: \"kubernetes.io/projected/78597359-922e-49db-b7e1-5fab26dbb467-tls-assets\") pod \"78597359-922e-49db-b7e1-5fab26dbb467\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") "
Oct 30 04:14:35.606604 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.606596    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78597359-922e-49db-b7e1-5fab26dbb467-kube-api-access-hrrnh\" (UniqueName: \"kubernetes.io/projected/78597359-922e-49db-b7e1-5fab26dbb467-kube-api-access-hrrnh\") pod \"78597359-922e-49db-b7e1-5fab26dbb467\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") "
Oct 30 04:14:35.606872 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.606622    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-config\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-config\") pod \"78597359-922e-49db-b7e1-5fab26dbb467\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") "
Oct 30 04:14:35.606872 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.606656    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config\") pod \"78597359-922e-49db-b7e1-5fab26dbb467\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") "
Oct 30 04:14:35.606872 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.606715    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-thanos-prometheus-http-client-file\") pod \"78597359-922e-49db-b7e1-5fab26dbb467\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") "
Oct 30 04:14:35.606872 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.606774    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"78597359-922e-49db-b7e1-5fab26dbb467\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") "
Oct 30 04:14:35.606872 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.606830    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"78597359-922e-49db-b7e1-5fab26dbb467\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") "
Oct 30 04:14:35.606872 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.606859    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-secret-prometheus-web-tls-ca\") pod \"78597359-922e-49db-b7e1-5fab26dbb467\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") "
Oct 30 04:14:35.607361 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.606894    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"78597359-922e-49db-b7e1-5fab26dbb467\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") "
Oct 30 04:14:35.607361 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.606926    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"78597359-922e-49db-b7e1-5fab26dbb467\" (UID: \"78597359-922e-49db-b7e1-5fab26dbb467\") "
Oct 30 04:14:35.609193 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.609122    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-2" pod "78597359-922e-49db-b7e1-5fab26dbb467" (UID: "78597359-922e-49db-b7e1-5fab26dbb467"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:14:35.609565 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.609393    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78597359-922e-49db-b7e1-5fab26dbb467-config-out" pod "78597359-922e-49db-b7e1-5fab26dbb467" (UID: "78597359-922e-49db-b7e1-5fab26dbb467"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:14:35.609565 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.609536    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-0" pod "78597359-922e-49db-b7e1-5fab26dbb467" (UID: "78597359-922e-49db-b7e1-5fab26dbb467"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:14:35.609881 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.609844    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-1" pod "78597359-922e-49db-b7e1-5fab26dbb467" (UID: "78597359-922e-49db-b7e1-5fab26dbb467"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:14:35.610233 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.610167    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "78597359-922e-49db-b7e1-5fab26dbb467" (UID: "78597359-922e-49db-b7e1-5fab26dbb467"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:35.610867 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.610797    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78597359-922e-49db-b7e1-5fab26dbb467-kube-api-access-hrrnh" pod "78597359-922e-49db-b7e1-5fab26dbb467" (UID: "78597359-922e-49db-b7e1-5fab26dbb467"). InnerVolumeSpecName "kube-api-access-hrrnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:14:35.612569 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.612518    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "78597359-922e-49db-b7e1-5fab26dbb467" (UID: "78597359-922e-49db-b7e1-5fab26dbb467"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:35.612808 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.612755    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-thanos-prometheus-http-client-file" pod "78597359-922e-49db-b7e1-5fab26dbb467" (UID: "78597359-922e-49db-b7e1-5fab26dbb467"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:35.613151 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.613110    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78597359-922e-49db-b7e1-5fab26dbb467-tls-assets" pod "78597359-922e-49db-b7e1-5fab26dbb467" (UID: "78597359-922e-49db-b7e1-5fab26dbb467"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:14:35.613784 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.613760    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87" pod "78597359-922e-49db-b7e1-5fab26dbb467" (UID: "78597359-922e-49db-b7e1-5fab26dbb467"). InnerVolumeSpecName "pvc-9689c530-7f91-4584-b28e-8809bed9e799". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:14:35.614265 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.614238    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-config" pod "78597359-922e-49db-b7e1-5fab26dbb467" (UID: "78597359-922e-49db-b7e1-5fab26dbb467"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:35.614402 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.614385    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-secret-prometheus-web-tls-ca" pod "78597359-922e-49db-b7e1-5fab26dbb467" (UID: "78597359-922e-49db-b7e1-5fab26dbb467"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:35.626483 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.626460    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config" pod "78597359-922e-49db-b7e1-5fab26dbb467" (UID: "78597359-922e-49db-b7e1-5fab26dbb467"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:35.635366 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.635348    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:35.707447 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.707408    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf88a46a-5476-4ef1-8266-a42af9af6c59-kube-api-access-xtchw\" (UniqueName: \"kubernetes.io/projected/bf88a46a-5476-4ef1-8266-a42af9af6c59-kube-api-access-xtchw\") pod \"bf88a46a-5476-4ef1-8266-a42af9af6c59\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") "
Oct 30 04:14:35.708058 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.707456    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-cluster-tls-config\") pod \"bf88a46a-5476-4ef1-8266-a42af9af6c59\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") "
Oct 30 04:14:35.708058 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.707481    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf88a46a-5476-4ef1-8266-a42af9af6c59-tls-assets\" (UniqueName: \"kubernetes.io/projected/bf88a46a-5476-4ef1-8266-a42af9af6c59-tls-assets\") pod \"bf88a46a-5476-4ef1-8266-a42af9af6c59\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") "
Oct 30 04:14:35.708058 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.707510    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf88a46a-5476-4ef1-8266-a42af9af6c59-config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf88a46a-5476-4ef1-8266-a42af9af6c59-config-out\") pod \"bf88a46a-5476-4ef1-8266-a42af9af6c59\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") "
Oct 30 04:14:35.708058 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.707537    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-web-config\" (UniqueName: \"kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-web-config\") pod \"bf88a46a-5476-4ef1-8266-a42af9af6c59\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") "
Oct 30 04:14:35.708058 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.707599    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf88a46a-5476-4ef1-8266-a42af9af6c59-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/bf88a46a-5476-4ef1-8266-a42af9af6c59-alertmanager-data-science-monitoringstack-db\") pod \"bf88a46a-5476-4ef1-8266-a42af9af6c59\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") "
Oct 30 04:14:35.708058 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.707641    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-config-volume\" (UniqueName: \"kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-config-volume\") pod \"bf88a46a-5476-4ef1-8266-a42af9af6c59\" (UID: \"bf88a46a-5476-4ef1-8266-a42af9af6c59\") "
Oct 30 04:14:35.708058 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.707959    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrrnh\" (UniqueName: \"kubernetes.io/projected/78597359-922e-49db-b7e1-5fab26dbb467-kube-api-access-hrrnh\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.708669 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.708377    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.708669 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.708406    4960 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.708669 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.708423    4960 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.708669 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.708439    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.708669 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.708455    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.708669 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.708469    4960 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-secret-prometheus-web-tls-ca\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.708669 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.708467    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf88a46a-5476-4ef1-8266-a42af9af6c59-alertmanager-data-science-monitoringstack-db" pod "bf88a46a-5476-4ef1-8266-a42af9af6c59" (UID: "bf88a46a-5476-4ef1-8266-a42af9af6c59"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:14:35.708669 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.708484    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/78597359-922e-49db-b7e1-5fab26dbb467-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.708669 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.708500    4960 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.708669 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.708530    4960 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") on node \"ip-10-0-128-241.ec2.internal\" "
Oct 30 04:14:35.708669 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.708546    4960 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78597359-922e-49db-b7e1-5fab26dbb467-config-out\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.708669 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.708562    4960 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/78597359-922e-49db-b7e1-5fab26dbb467-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.708669 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.708608    4960 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78597359-922e-49db-b7e1-5fab26dbb467-tls-assets\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.710474 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.710446    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf88a46a-5476-4ef1-8266-a42af9af6c59-config-out" pod "bf88a46a-5476-4ef1-8266-a42af9af6c59" (UID: "bf88a46a-5476-4ef1-8266-a42af9af6c59"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:14:35.710623 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.710603    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf88a46a-5476-4ef1-8266-a42af9af6c59-kube-api-access-xtchw" pod "bf88a46a-5476-4ef1-8266-a42af9af6c59" (UID: "bf88a46a-5476-4ef1-8266-a42af9af6c59"). InnerVolumeSpecName "kube-api-access-xtchw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:14:35.710782 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.710752    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf88a46a-5476-4ef1-8266-a42af9af6c59-tls-assets" pod "bf88a46a-5476-4ef1-8266-a42af9af6c59" (UID: "bf88a46a-5476-4ef1-8266-a42af9af6c59"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:14:35.710852 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.710815    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-config-volume" pod "bf88a46a-5476-4ef1-8266-a42af9af6c59" (UID: "bf88a46a-5476-4ef1-8266-a42af9af6c59"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:35.714922 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.714898    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-cluster-tls-config" pod "bf88a46a-5476-4ef1-8266-a42af9af6c59" (UID: "bf88a46a-5476-4ef1-8266-a42af9af6c59"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:35.720916 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.720901    4960 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-9689c530-7f91-4584-b28e-8809bed9e799" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87") on node "ip-10-0-128-241.ec2.internal"
Oct 30 04:14:35.722013 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.721995    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-web-config" pod "bf88a46a-5476-4ef1-8266-a42af9af6c59" (UID: "bf88a46a-5476-4ef1-8266-a42af9af6c59"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:35.810893 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.810870    4960 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/bf88a46a-5476-4ef1-8266-a42af9af6c59-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.810996 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.810896    4960 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-config-volume\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.810996 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.810911    4960 reconciler_common.go:299] "Volume detached for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"csi-6f8c8e4c1326657251c30594036033694cadf25e97408e4f9d9da436f78f3fa4\""
Oct 30 04:14:35.810996 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.810928    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtchw\" (UniqueName: \"kubernetes.io/projected/bf88a46a-5476-4ef1-8266-a42af9af6c59-kube-api-access-xtchw\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.810996 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.810942    4960 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-cluster-tls-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.810996 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.810956    4960 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf88a46a-5476-4ef1-8266-a42af9af6c59-tls-assets\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.810996 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.810971    4960 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf88a46a-5476-4ef1-8266-a42af9af6c59-config-out\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.810996 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.810984    4960 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf88a46a-5476-4ef1-8266-a42af9af6c59-web-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.878717 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.878498    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:35.881264 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.881232    4960 generic.go:356] "Generic (PLEG): container finished" podID="78597359-922e-49db-b7e1-5fab26dbb467" containerID="fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76" exitCode=0
Oct 30 04:14:35.881264 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.881261    4960 generic.go:356] "Generic (PLEG): container finished" podID="78597359-922e-49db-b7e1-5fab26dbb467" containerID="e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149" exitCode=0
Oct 30 04:14:35.881452 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.881272    4960 generic.go:356] "Generic (PLEG): container finished" podID="78597359-922e-49db-b7e1-5fab26dbb467" containerID="940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88" exitCode=0
Oct 30 04:14:35.881452 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.881306    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"78597359-922e-49db-b7e1-5fab26dbb467","Type":"ContainerDied","Data":"fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76"}
Oct 30 04:14:35.881452 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.881312    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:35.881452 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.881373    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"78597359-922e-49db-b7e1-5fab26dbb467","Type":"ContainerDied","Data":"e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149"}
Oct 30 04:14:35.881452 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.881414    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"78597359-922e-49db-b7e1-5fab26dbb467","Type":"ContainerDied","Data":"940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88"}
Oct 30 04:14:35.881452 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.881454    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"78597359-922e-49db-b7e1-5fab26dbb467","Type":"ContainerDied","Data":"58183ff5860c57a0f275419124d55bd4d34162c0554cf2fbf11b0529744157f3"}
Oct 30 04:14:35.881760 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.881464    4960 scope.go:122] "RemoveContainer" containerID="fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76"
Oct 30 04:14:35.883161 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.883140    4960 generic.go:356] "Generic (PLEG): container finished" podID="93a0812d-ed41-455f-8371-0409cad2b544" containerID="38b3eadc37d9e6dda2bfdca81e1b641751feac5ee1a77432343a8f05f6bf0db9" exitCode=0
Oct 30 04:14:35.883903 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.883255    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:35.883903 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.883276    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"93a0812d-ed41-455f-8371-0409cad2b544","Type":"ContainerDied","Data":"38b3eadc37d9e6dda2bfdca81e1b641751feac5ee1a77432343a8f05f6bf0db9"}
Oct 30 04:14:35.883903 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.883313    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"93a0812d-ed41-455f-8371-0409cad2b544","Type":"ContainerDied","Data":"bca590c2d5080e1aef6ae2dbe11784b11b7f441a5c866a6a3a7f9daf4297b353"}
Oct 30 04:14:35.888074 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.888051    4960 generic.go:356] "Generic (PLEG): container finished" podID="bf88a46a-5476-4ef1-8266-a42af9af6c59" containerID="90de8c5bc87574cd83039d70bfd5256e0c1424d783ecc528029bb81b478e2d23" exitCode=0
Oct 30 04:14:35.888074 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.888075    4960 generic.go:356] "Generic (PLEG): container finished" podID="bf88a46a-5476-4ef1-8266-a42af9af6c59" containerID="9c27e3910d90b3c03d6db0c9ccbfeb841dc67132f0f8e02649495a4f9160c219" exitCode=0
Oct 30 04:14:35.888289 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.888111    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"bf88a46a-5476-4ef1-8266-a42af9af6c59","Type":"ContainerDied","Data":"90de8c5bc87574cd83039d70bfd5256e0c1424d783ecc528029bb81b478e2d23"}
Oct 30 04:14:35.888289 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.888158    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"bf88a46a-5476-4ef1-8266-a42af9af6c59","Type":"ContainerDied","Data":"9c27e3910d90b3c03d6db0c9ccbfeb841dc67132f0f8e02649495a4f9160c219"}
Oct 30 04:14:35.888289 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.888220    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"bf88a46a-5476-4ef1-8266-a42af9af6c59","Type":"ContainerDied","Data":"583cfaad5cf7d58cb8481aa7014fb3d56ff9ac401a4b43297c23cbc85236d220"}
Oct 30 04:14:35.888471 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.888303    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:35.892351 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.892329    4960 scope.go:122] "RemoveContainer" containerID="e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149"
Oct 30 04:14:35.903268 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.903238    4960 scope.go:122] "RemoveContainer" containerID="940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88"
Oct 30 04:14:35.911980 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.911960    4960 scope.go:122] "RemoveContainer" containerID="39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43"
Oct 30 04:14:35.923006 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.922990    4960 scope.go:122] "RemoveContainer" containerID="fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76"
Oct 30 04:14:35.923349 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:14:35.923319    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76\": container with ID starting with fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76 not found: ID does not exist" containerID="fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76"
Oct 30 04:14:35.923424 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.923357    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76"} err="failed to get container status \"fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76\": rpc error: code = NotFound desc = could not find container \"fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76\": container with ID starting with fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76 not found: ID does not exist"
Oct 30 04:14:35.923424 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.923375    4960 scope.go:122] "RemoveContainer" containerID="e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149"
Oct 30 04:14:35.923699 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:14:35.923678    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149\": container with ID starting with e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149 not found: ID does not exist" containerID="e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149"
Oct 30 04:14:35.923787 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.923707    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149"} err="failed to get container status \"e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149\": rpc error: code = NotFound desc = could not find container \"e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149\": container with ID starting with e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149 not found: ID does not exist"
Oct 30 04:14:35.923787 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.923732    4960 scope.go:122] "RemoveContainer" containerID="940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88"
Oct 30 04:14:35.924082 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:14:35.924059    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88\": container with ID starting with 940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88 not found: ID does not exist" containerID="940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88"
Oct 30 04:14:35.924128 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.924094    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88"} err="failed to get container status \"940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88\": rpc error: code = NotFound desc = could not find container \"940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88\": container with ID starting with 940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88 not found: ID does not exist"
Oct 30 04:14:35.924128 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.924117    4960 scope.go:122] "RemoveContainer" containerID="39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43"
Oct 30 04:14:35.924418 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:14:35.924397    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43\": container with ID starting with 39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43 not found: ID does not exist" containerID="39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43"
Oct 30 04:14:35.924472 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.924425    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43"} err="failed to get container status \"39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43\": rpc error: code = NotFound desc = could not find container \"39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43\": container with ID starting with 39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43 not found: ID does not exist"
Oct 30 04:14:35.924472 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.924445    4960 scope.go:122] "RemoveContainer" containerID="fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76"
Oct 30 04:14:35.924695 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.924676    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76"} err="failed to get container status \"fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76\": rpc error: code = NotFound desc = could not find container \"fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76\": container with ID starting with fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76 not found: ID does not exist"
Oct 30 04:14:35.924769 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.924698    4960 scope.go:122] "RemoveContainer" containerID="e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149"
Oct 30 04:14:35.924951 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.924931    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149"} err="failed to get container status \"e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149\": rpc error: code = NotFound desc = could not find container \"e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149\": container with ID starting with e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149 not found: ID does not exist"
Oct 30 04:14:35.925006 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.924952    4960 scope.go:122] "RemoveContainer" containerID="940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88"
Oct 30 04:14:35.925252 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.925228    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88"} err="failed to get container status \"940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88\": rpc error: code = NotFound desc = could not find container \"940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88\": container with ID starting with 940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88 not found: ID does not exist"
Oct 30 04:14:35.925344 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.925254    4960 scope.go:122] "RemoveContainer" containerID="39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43"
Oct 30 04:14:35.925581 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.925560    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43"} err="failed to get container status \"39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43\": rpc error: code = NotFound desc = could not find container \"39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43\": container with ID starting with 39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43 not found: ID does not exist"
Oct 30 04:14:35.925657 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.925583    4960 scope.go:122] "RemoveContainer" containerID="fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76"
Oct 30 04:14:35.925856 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.925838    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76"} err="failed to get container status \"fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76\": rpc error: code = NotFound desc = could not find container \"fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76\": container with ID starting with fb25e4fa7f13144377e6c0bc58972bcd19e80b0d7b80b994e8d9dc9c6ab07a76 not found: ID does not exist"
Oct 30 04:14:35.925923 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.925857    4960 scope.go:122] "RemoveContainer" containerID="e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149"
Oct 30 04:14:35.926108 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.926082    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149"} err="failed to get container status \"e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149\": rpc error: code = NotFound desc = could not find container \"e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149\": container with ID starting with e02bdca891e88c6e676ab2fdb0fa35c41280d97666b37c2506ee294be9f44149 not found: ID does not exist"
Oct 30 04:14:35.926169 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.926111    4960 scope.go:122] "RemoveContainer" containerID="940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88"
Oct 30 04:14:35.926429 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.926405    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88"} err="failed to get container status \"940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88\": rpc error: code = NotFound desc = could not find container \"940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88\": container with ID starting with 940e4cd9bda9721efe676d303bb40e48c48766b2e8531c328fbaa5f92ab37c88 not found: ID does not exist"
Oct 30 04:14:35.926485 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.926432    4960 scope.go:122] "RemoveContainer" containerID="39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43"
Oct 30 04:14:35.926987 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.926960    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43"} err="failed to get container status \"39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43\": rpc error: code = NotFound desc = could not find container \"39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43\": container with ID starting with 39f4afef5b0cff3ee3f6253cef3a162d8a4f7e453fc9e6834b59be68417e7b43 not found: ID does not exist"
Oct 30 04:14:35.927069 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.926987    4960 scope.go:122] "RemoveContainer" containerID="38b3eadc37d9e6dda2bfdca81e1b641751feac5ee1a77432343a8f05f6bf0db9"
Oct 30 04:14:35.935675 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.935642    4960 scope.go:122] "RemoveContainer" containerID="38b3eadc37d9e6dda2bfdca81e1b641751feac5ee1a77432343a8f05f6bf0db9"
Oct 30 04:14:35.935963 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:14:35.935929    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b3eadc37d9e6dda2bfdca81e1b641751feac5ee1a77432343a8f05f6bf0db9\": container with ID starting with 38b3eadc37d9e6dda2bfdca81e1b641751feac5ee1a77432343a8f05f6bf0db9 not found: ID does not exist" containerID="38b3eadc37d9e6dda2bfdca81e1b641751feac5ee1a77432343a8f05f6bf0db9"
Oct 30 04:14:35.936019 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.935972    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b3eadc37d9e6dda2bfdca81e1b641751feac5ee1a77432343a8f05f6bf0db9"} err="failed to get container status \"38b3eadc37d9e6dda2bfdca81e1b641751feac5ee1a77432343a8f05f6bf0db9\": rpc error: code = NotFound desc = could not find container \"38b3eadc37d9e6dda2bfdca81e1b641751feac5ee1a77432343a8f05f6bf0db9\": container with ID starting with 38b3eadc37d9e6dda2bfdca81e1b641751feac5ee1a77432343a8f05f6bf0db9 not found: ID does not exist"
Oct 30 04:14:35.936019 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.935991    4960 scope.go:122] "RemoveContainer" containerID="90de8c5bc87574cd83039d70bfd5256e0c1424d783ecc528029bb81b478e2d23"
Oct 30 04:14:35.942309 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.942287    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:14:35.945139 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.945113    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:14:35.945574 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.945558    4960 scope.go:122] "RemoveContainer" containerID="9c27e3910d90b3c03d6db0c9ccbfeb841dc67132f0f8e02649495a4f9160c219"
Oct 30 04:14:35.954472 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.954449    4960 scope.go:122] "RemoveContainer" containerID="cdaade17437bfffe3fbbdfbfa667a39cf5419e5b96c60d3bee08e36209bf2a9f"
Oct 30 04:14:35.963555 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.963539    4960 scope.go:122] "RemoveContainer" containerID="90de8c5bc87574cd83039d70bfd5256e0c1424d783ecc528029bb81b478e2d23"
Oct 30 04:14:35.963836 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:14:35.963799    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90de8c5bc87574cd83039d70bfd5256e0c1424d783ecc528029bb81b478e2d23\": container with ID starting with 90de8c5bc87574cd83039d70bfd5256e0c1424d783ecc528029bb81b478e2d23 not found: ID does not exist" containerID="90de8c5bc87574cd83039d70bfd5256e0c1424d783ecc528029bb81b478e2d23"
Oct 30 04:14:35.963892 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.963848    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90de8c5bc87574cd83039d70bfd5256e0c1424d783ecc528029bb81b478e2d23"} err="failed to get container status \"90de8c5bc87574cd83039d70bfd5256e0c1424d783ecc528029bb81b478e2d23\": rpc error: code = NotFound desc = could not find container \"90de8c5bc87574cd83039d70bfd5256e0c1424d783ecc528029bb81b478e2d23\": container with ID starting with 90de8c5bc87574cd83039d70bfd5256e0c1424d783ecc528029bb81b478e2d23 not found: ID does not exist"
Oct 30 04:14:35.963892 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.963873    4960 scope.go:122] "RemoveContainer" containerID="9c27e3910d90b3c03d6db0c9ccbfeb841dc67132f0f8e02649495a4f9160c219"
Oct 30 04:14:35.964095 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:14:35.964078    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c27e3910d90b3c03d6db0c9ccbfeb841dc67132f0f8e02649495a4f9160c219\": container with ID starting with 9c27e3910d90b3c03d6db0c9ccbfeb841dc67132f0f8e02649495a4f9160c219 not found: ID does not exist" containerID="9c27e3910d90b3c03d6db0c9ccbfeb841dc67132f0f8e02649495a4f9160c219"
Oct 30 04:14:35.964139 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.964102    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c27e3910d90b3c03d6db0c9ccbfeb841dc67132f0f8e02649495a4f9160c219"} err="failed to get container status \"9c27e3910d90b3c03d6db0c9ccbfeb841dc67132f0f8e02649495a4f9160c219\": rpc error: code = NotFound desc = could not find container \"9c27e3910d90b3c03d6db0c9ccbfeb841dc67132f0f8e02649495a4f9160c219\": container with ID starting with 9c27e3910d90b3c03d6db0c9ccbfeb841dc67132f0f8e02649495a4f9160c219 not found: ID does not exist"
Oct 30 04:14:35.964139 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.964115    4960 scope.go:122] "RemoveContainer" containerID="cdaade17437bfffe3fbbdfbfa667a39cf5419e5b96c60d3bee08e36209bf2a9f"
Oct 30 04:14:35.964340 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:14:35.964320    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdaade17437bfffe3fbbdfbfa667a39cf5419e5b96c60d3bee08e36209bf2a9f\": container with ID starting with cdaade17437bfffe3fbbdfbfa667a39cf5419e5b96c60d3bee08e36209bf2a9f not found: ID does not exist" containerID="cdaade17437bfffe3fbbdfbfa667a39cf5419e5b96c60d3bee08e36209bf2a9f"
Oct 30 04:14:35.964385 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.964345    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdaade17437bfffe3fbbdfbfa667a39cf5419e5b96c60d3bee08e36209bf2a9f"} err="failed to get container status \"cdaade17437bfffe3fbbdfbfa667a39cf5419e5b96c60d3bee08e36209bf2a9f\": rpc error: code = NotFound desc = could not find container \"cdaade17437bfffe3fbbdfbfa667a39cf5419e5b96c60d3bee08e36209bf2a9f\": container with ID starting with cdaade17437bfffe3fbbdfbfa667a39cf5419e5b96c60d3bee08e36209bf2a9f not found: ID does not exist"
Oct 30 04:14:35.964385 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.964360    4960 scope.go:122] "RemoveContainer" containerID="90de8c5bc87574cd83039d70bfd5256e0c1424d783ecc528029bb81b478e2d23"
Oct 30 04:14:35.964585 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.964564    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90de8c5bc87574cd83039d70bfd5256e0c1424d783ecc528029bb81b478e2d23"} err="failed to get container status \"90de8c5bc87574cd83039d70bfd5256e0c1424d783ecc528029bb81b478e2d23\": rpc error: code = NotFound desc = could not find container \"90de8c5bc87574cd83039d70bfd5256e0c1424d783ecc528029bb81b478e2d23\": container with ID starting with 90de8c5bc87574cd83039d70bfd5256e0c1424d783ecc528029bb81b478e2d23 not found: ID does not exist"
Oct 30 04:14:35.964658 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.964587    4960 scope.go:122] "RemoveContainer" containerID="9c27e3910d90b3c03d6db0c9ccbfeb841dc67132f0f8e02649495a4f9160c219"
Oct 30 04:14:35.964920 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.964889    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c27e3910d90b3c03d6db0c9ccbfeb841dc67132f0f8e02649495a4f9160c219"} err="failed to get container status \"9c27e3910d90b3c03d6db0c9ccbfeb841dc67132f0f8e02649495a4f9160c219\": rpc error: code = NotFound desc = could not find container \"9c27e3910d90b3c03d6db0c9ccbfeb841dc67132f0f8e02649495a4f9160c219\": container with ID starting with 9c27e3910d90b3c03d6db0c9ccbfeb841dc67132f0f8e02649495a4f9160c219 not found: ID does not exist"
Oct 30 04:14:35.964920 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.964921    4960 scope.go:122] "RemoveContainer" containerID="cdaade17437bfffe3fbbdfbfa667a39cf5419e5b96c60d3bee08e36209bf2a9f"
Oct 30 04:14:35.965258 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.965237    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdaade17437bfffe3fbbdfbfa667a39cf5419e5b96c60d3bee08e36209bf2a9f"} err="failed to get container status \"cdaade17437bfffe3fbbdfbfa667a39cf5419e5b96c60d3bee08e36209bf2a9f\": rpc error: code = NotFound desc = could not find container \"cdaade17437bfffe3fbbdfbfa667a39cf5419e5b96c60d3bee08e36209bf2a9f\": container with ID starting with cdaade17437bfffe3fbbdfbfa667a39cf5419e5b96c60d3bee08e36209bf2a9f not found: ID does not exist"
Oct 30 04:14:35.984254 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.984226    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:14:35.987121 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.987099    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:14:35.988351 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:35.988329    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:36.025374 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:36.025357    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:14:36.029279 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:36.029260    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:14:36.092929 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:36.092906    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:36.816915 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:36.816882    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:36.946937 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:36.946908    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:37.074224 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:37.074124    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:37.502455 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:37.502381    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78597359-922e-49db-b7e1-5fab26dbb467" path="/var/lib/kubelet/pods/78597359-922e-49db-b7e1-5fab26dbb467/volumes"
Oct 30 04:14:37.502922 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:37.502905    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a0812d-ed41-455f-8371-0409cad2b544" path="/var/lib/kubelet/pods/93a0812d-ed41-455f-8371-0409cad2b544/volumes"
Oct 30 04:14:37.503552 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:37.503535    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf88a46a-5476-4ef1-8266-a42af9af6c59" path="/var/lib/kubelet/pods/bf88a46a-5476-4ef1-8266-a42af9af6c59/volumes"
Oct 30 04:14:41.149893 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.149857    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:41.260429 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.260395    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:41.375696 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.375662    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:41.438746 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.438663    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"]
Oct 30 04:14:41.438989 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.438976    4960 cpu_manager.go:414] "containerMap: removing container" podUID="bf88a46a-5476-4ef1-8266-a42af9af6c59" containerName="init-config-reloader"
Oct 30 04:14:41.439041 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.438990    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf88a46a-5476-4ef1-8266-a42af9af6c59" containerName="init-config-reloader"
Oct 30 04:14:41.439041 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439005    4960 cpu_manager.go:414] "containerMap: removing container" podUID="bf88a46a-5476-4ef1-8266-a42af9af6c59" containerName="alertmanager"
Oct 30 04:14:41.439041 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439011    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf88a46a-5476-4ef1-8266-a42af9af6c59" containerName="alertmanager"
Oct 30 04:14:41.439041 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439021    4960 cpu_manager.go:414] "containerMap: removing container" podUID="bf88a46a-5476-4ef1-8266-a42af9af6c59" containerName="config-reloader"
Oct 30 04:14:41.439041 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439027    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf88a46a-5476-4ef1-8266-a42af9af6c59" containerName="config-reloader"
Oct 30 04:14:41.439041 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439033    4960 cpu_manager.go:414] "containerMap: removing container" podUID="78597359-922e-49db-b7e1-5fab26dbb467" containerName="config-reloader"
Oct 30 04:14:41.439041 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439038    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78597359-922e-49db-b7e1-5fab26dbb467" containerName="config-reloader"
Oct 30 04:14:41.439041 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439043    4960 cpu_manager.go:414] "containerMap: removing container" podUID="78597359-922e-49db-b7e1-5fab26dbb467" containerName="thanos-sidecar"
Oct 30 04:14:41.439297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439048    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78597359-922e-49db-b7e1-5fab26dbb467" containerName="thanos-sidecar"
Oct 30 04:14:41.439297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439056    4960 cpu_manager.go:414] "containerMap: removing container" podUID="78597359-922e-49db-b7e1-5fab26dbb467" containerName="prometheus"
Oct 30 04:14:41.439297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439061    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78597359-922e-49db-b7e1-5fab26dbb467" containerName="prometheus"
Oct 30 04:14:41.439297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439067    4960 cpu_manager.go:414] "containerMap: removing container" podUID="19b85040-196e-468c-939d-8b93b291d593" containerName="korrel8r"
Oct 30 04:14:41.439297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439072    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b85040-196e-468c-939d-8b93b291d593" containerName="korrel8r"
Oct 30 04:14:41.439297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439084    4960 cpu_manager.go:414] "containerMap: removing container" podUID="2d5782f1-d219-41cf-addd-375dd8c3cb1d" containerName="kube-rbac-proxy"
Oct 30 04:14:41.439297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439089    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d5782f1-d219-41cf-addd-375dd8c3cb1d" containerName="kube-rbac-proxy"
Oct 30 04:14:41.439297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439095    4960 cpu_manager.go:414] "containerMap: removing container" podUID="78597359-922e-49db-b7e1-5fab26dbb467" containerName="init-config-reloader"
Oct 30 04:14:41.439297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439100    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78597359-922e-49db-b7e1-5fab26dbb467" containerName="init-config-reloader"
Oct 30 04:14:41.439297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439108    4960 cpu_manager.go:414] "containerMap: removing container" podUID="93a0812d-ed41-455f-8371-0409cad2b544" containerName="otc-container"
Oct 30 04:14:41.439297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439112    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a0812d-ed41-455f-8371-0409cad2b544" containerName="otc-container"
Oct 30 04:14:41.439297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439159    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf88a46a-5476-4ef1-8266-a42af9af6c59" containerName="alertmanager"
Oct 30 04:14:41.439297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439168    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf88a46a-5476-4ef1-8266-a42af9af6c59" containerName="config-reloader"
Oct 30 04:14:41.439297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439173    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="78597359-922e-49db-b7e1-5fab26dbb467" containerName="config-reloader"
Oct 30 04:14:41.439297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439192    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="78597359-922e-49db-b7e1-5fab26dbb467" containerName="thanos-sidecar"
Oct 30 04:14:41.439297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439198    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="78597359-922e-49db-b7e1-5fab26dbb467" containerName="prometheus"
Oct 30 04:14:41.439297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439203    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b85040-196e-468c-939d-8b93b291d593" containerName="korrel8r"
Oct 30 04:14:41.439297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439210    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d5782f1-d219-41cf-addd-375dd8c3cb1d" containerName="kube-rbac-proxy"
Oct 30 04:14:41.439297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.439216    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a0812d-ed41-455f-8371-0409cad2b544" containerName="otc-container"
Oct 30 04:14:41.443807 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.443789    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:14:41.460101 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.460079    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"]
Oct 30 04:14:41.461638 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.460793    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 04:14:41.461638 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.460800    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\""
Oct 30 04:14:41.461638 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.460818    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:14:41.461638 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.460821    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\""
Oct 30 04:14:41.461638 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.460916    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:14:41.461638 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.460923    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-dx2dx\""
Oct 30 04:14:41.461638 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.460926    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:14:41.480076 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.480055    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"]
Oct 30 04:14:41.483605 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.483584    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"
Oct 30 04:14:41.491812 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.491790    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:41.505104 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.505083    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"]
Oct 30 04:14:41.521349 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.521327    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b"]
Oct 30 04:14:41.525418 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.525398    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b"
Oct 30 04:14:41.558225 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.551833    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-s424b\" (UID: \"ae264e2d-42ab-435c-a4f4-df406a2c9ee8\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b"
Oct 30 04:14:41.558225 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.551878    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/73fbee4b-fa9b-47e7-b350-427e832dd16c-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-sfftq\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"
Oct 30 04:14:41.558225 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.551918    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-c822n\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:14:41.558225 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.551982    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-sfftq\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"
Oct 30 04:14:41.558225 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.552006    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-c822n\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:14:41.558225 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.552029    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4gff\" (UniqueName: \"kubernetes.io/projected/02b485e3-c53a-4126-8854-1639da65cb05-kube-api-access-p4gff\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-c822n\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:14:41.558225 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.552064    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-sfftq\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"
Oct 30 04:14:41.558225 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.552122    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-s424b\" (UID: \"ae264e2d-42ab-435c-a4f4-df406a2c9ee8\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b"
Oct 30 04:14:41.558225 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.552156    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/02b485e3-c53a-4126-8854-1639da65cb05-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-c822n\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:14:41.558225 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.552202    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-sfftq\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"
Oct 30 04:14:41.558225 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.552226    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr7w9\" (UniqueName: \"kubernetes.io/projected/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-kube-api-access-cr7w9\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-s424b\" (UID: \"ae264e2d-42ab-435c-a4f4-df406a2c9ee8\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b"
Oct 30 04:14:41.558225 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.552253    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6wsc\" (UniqueName: \"kubernetes.io/projected/73fbee4b-fa9b-47e7-b350-427e832dd16c-kube-api-access-c6wsc\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-sfftq\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"
Oct 30 04:14:41.558225 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.552304    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-c822n\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:14:41.558225 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.553436    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b"]
Oct 30 04:14:41.558225 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.557146    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"]
Oct 30 04:14:41.560914 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.560889    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\""
Oct 30 04:14:41.561041 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.560937    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-tnt4d\""
Oct 30 04:14:41.561041 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.560950    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\""
Oct 30 04:14:41.561041 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.560991    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\""
Oct 30 04:14:41.561195 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.561086    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-n4dcg\""
Oct 30 04:14:41.561195 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.561102    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\""
Oct 30 04:14:41.561356 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.561334    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\""
Oct 30 04:14:41.561794 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.561777    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"
Oct 30 04:14:41.583841 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.583690    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"]
Oct 30 04:14:41.651777 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.651752    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\""
Oct 30 04:14:41.651777 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.651752    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-q6k89\""
Oct 30 04:14:41.653014 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.652996    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/07b685f5-e45c-4c9c-a170-6d26a643341b-tls\") pod \"korrel8r-5cbcc89b8f-xnsgw\" (UID: \"07b685f5-e45c-4c9c-a170-6d26a643341b\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"
Oct 30 04:14:41.653076 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.653026    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-s424b\" (UID: \"ae264e2d-42ab-435c-a4f4-df406a2c9ee8\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b"
Oct 30 04:14:41.653076 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.653048    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/02b485e3-c53a-4126-8854-1639da65cb05-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-c822n\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:14:41.653076 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.653065    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-sfftq\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"
Oct 30 04:14:41.653171 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.653127    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr7w9\" (UniqueName: \"kubernetes.io/projected/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-kube-api-access-cr7w9\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-s424b\" (UID: \"ae264e2d-42ab-435c-a4f4-df406a2c9ee8\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b"
Oct 30 04:14:41.653171 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.653156    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6wsc\" (UniqueName: \"kubernetes.io/projected/73fbee4b-fa9b-47e7-b350-427e832dd16c-kube-api-access-c6wsc\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-sfftq\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"
Oct 30 04:14:41.653264 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.653219    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf8kw\" (UniqueName: \"kubernetes.io/projected/07b685f5-e45c-4c9c-a170-6d26a643341b-kube-api-access-tf8kw\") pod \"korrel8r-5cbcc89b8f-xnsgw\" (UID: \"07b685f5-e45c-4c9c-a170-6d26a643341b\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"
Oct 30 04:14:41.653264 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.653249    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-c822n\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:14:41.653360 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.653285    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-s424b\" (UID: \"ae264e2d-42ab-435c-a4f4-df406a2c9ee8\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b"
Oct 30 04:14:41.653423 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.653379    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/73fbee4b-fa9b-47e7-b350-427e832dd16c-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-sfftq\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"
Oct 30 04:14:41.653423 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.653415    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07b685f5-e45c-4c9c-a170-6d26a643341b-config\") pod \"korrel8r-5cbcc89b8f-xnsgw\" (UID: \"07b685f5-e45c-4c9c-a170-6d26a643341b\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"
Oct 30 04:14:41.653522 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.653458    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-c822n\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:14:41.653522 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.653511    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-sfftq\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"
Oct 30 04:14:41.653620 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.653555    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-c822n\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:14:41.653620 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.653577    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4gff\" (UniqueName: \"kubernetes.io/projected/02b485e3-c53a-4126-8854-1639da65cb05-kube-api-access-p4gff\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-c822n\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:14:41.653620 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.653613    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-sfftq\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"
Oct 30 04:14:41.654153 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.653952    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-sfftq\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"
Oct 30 04:14:41.654153 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.654135    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/02b485e3-c53a-4126-8854-1639da65cb05-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-c822n\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:14:41.654329 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.654233    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-sfftq\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"
Oct 30 04:14:41.654329 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.654276    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-sfftq\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"
Oct 30 04:14:41.654454 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.654354    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-s424b\" (UID: \"ae264e2d-42ab-435c-a4f4-df406a2c9ee8\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b"
Oct 30 04:14:41.656087 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.656064    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-c822n\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:14:41.656551 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.656526    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-c822n\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:14:41.656633 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.656551    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-s424b\" (UID: \"ae264e2d-42ab-435c-a4f4-df406a2c9ee8\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b"
Oct 30 04:14:41.656750 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.656735    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/73fbee4b-fa9b-47e7-b350-427e832dd16c-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-sfftq\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"
Oct 30 04:14:41.656887 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.656870    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-c822n\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:14:41.661653 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.661631    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\""
Oct 30 04:14:41.673582 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.673562    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr7w9\" (UniqueName: \"kubernetes.io/projected/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-kube-api-access-cr7w9\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-s424b\" (UID: \"ae264e2d-42ab-435c-a4f4-df406a2c9ee8\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b"
Oct 30 04:14:41.673676 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.673632    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4gff\" (UniqueName: \"kubernetes.io/projected/02b485e3-c53a-4126-8854-1639da65cb05-kube-api-access-p4gff\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-c822n\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:14:41.674029 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.674011    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6wsc\" (UniqueName: \"kubernetes.io/projected/73fbee4b-fa9b-47e7-b350-427e832dd16c-kube-api-access-c6wsc\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-sfftq\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"
Oct 30 04:14:41.754395 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.754344    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/07b685f5-e45c-4c9c-a170-6d26a643341b-tls\") pod \"korrel8r-5cbcc89b8f-xnsgw\" (UID: \"07b685f5-e45c-4c9c-a170-6d26a643341b\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"
Oct 30 04:14:41.754395 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.754383    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf8kw\" (UniqueName: \"kubernetes.io/projected/07b685f5-e45c-4c9c-a170-6d26a643341b-kube-api-access-tf8kw\") pod \"korrel8r-5cbcc89b8f-xnsgw\" (UID: \"07b685f5-e45c-4c9c-a170-6d26a643341b\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"
Oct 30 04:14:41.754550 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.754406    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07b685f5-e45c-4c9c-a170-6d26a643341b-config\") pod \"korrel8r-5cbcc89b8f-xnsgw\" (UID: \"07b685f5-e45c-4c9c-a170-6d26a643341b\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"
Oct 30 04:14:41.754550 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:14:41.754489    4960 secret.go:189] Couldn't get secret redhat-ods-monitoring/korrel8r-tls: secret "korrel8r-tls" not found
Oct 30 04:14:41.754665 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:14:41.754573    4960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07b685f5-e45c-4c9c-a170-6d26a643341b-tls podName:07b685f5-e45c-4c9c-a170-6d26a643341b nodeName:}" failed. No retries permitted until 2026-10-30 04:14:42.254553123 +0000 UTC m=+5562.903601525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/07b685f5-e45c-4c9c-a170-6d26a643341b-tls") pod "korrel8r-5cbcc89b8f-xnsgw" (UID: "07b685f5-e45c-4c9c-a170-6d26a643341b") : secret "korrel8r-tls" not found
Oct 30 04:14:41.755031 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.755014    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07b685f5-e45c-4c9c-a170-6d26a643341b-config\") pod \"korrel8r-5cbcc89b8f-xnsgw\" (UID: \"07b685f5-e45c-4c9c-a170-6d26a643341b\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"
Oct 30 04:14:41.756285 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.756270    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:14:41.776564 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.776545    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf8kw\" (UniqueName: \"kubernetes.io/projected/07b685f5-e45c-4c9c-a170-6d26a643341b-kube-api-access-tf8kw\") pod \"korrel8r-5cbcc89b8f-xnsgw\" (UID: \"07b685f5-e45c-4c9c-a170-6d26a643341b\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"
Oct 30 04:14:41.799311 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.799289    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"
Oct 30 04:14:41.862732 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.862531    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b"
Oct 30 04:14:41.919655 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:41.919619    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:42.048511 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.048488    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"]
Oct 30 04:14:42.049986 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:14:42.049959    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b485e3_c53a_4126_8854_1639da65cb05.slice/crio-2ac5d83507c1643358d21712e0e69eafffca5dc3f7b531bc121fe4bc71c1f7e4 WatchSource:0}: Error finding container 2ac5d83507c1643358d21712e0e69eafffca5dc3f7b531bc121fe4bc71c1f7e4: Status 404 returned error can't find the container with id 2ac5d83507c1643358d21712e0e69eafffca5dc3f7b531bc121fe4bc71c1f7e4
Oct 30 04:14:42.073413 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.073387    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:42.103719 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.103696    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"]
Oct 30 04:14:42.104876 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:14:42.104849    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73fbee4b_fa9b_47e7_b350_427e832dd16c.slice/crio-9d9405660fda43097585010c84f03b26cfcad339ffd0ca522b6f09846034f9e0 WatchSource:0}: Error finding container 9d9405660fda43097585010c84f03b26cfcad339ffd0ca522b6f09846034f9e0: Status 404 returned error can't find the container with id 9d9405660fda43097585010c84f03b26cfcad339ffd0ca522b6f09846034f9e0
Oct 30 04:14:42.164646 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.164626    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b"]
Oct 30 04:14:42.166172 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:14:42.166144    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae264e2d_42ab_435c_a4f4_df406a2c9ee8.slice/crio-541e18575ce5e324f0209e5899d83c1cec1e3d13e607f747f949985949b7a0ab WatchSource:0}: Error finding container 541e18575ce5e324f0209e5899d83c1cec1e3d13e607f747f949985949b7a0ab: Status 404 returned error can't find the container with id 541e18575ce5e324f0209e5899d83c1cec1e3d13e607f747f949985949b7a0ab
Oct 30 04:14:42.195864 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.195840    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:42.259266 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.259241    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/07b685f5-e45c-4c9c-a170-6d26a643341b-tls\") pod \"korrel8r-5cbcc89b8f-xnsgw\" (UID: \"07b685f5-e45c-4c9c-a170-6d26a643341b\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"
Oct 30 04:14:42.261633 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.261609    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/07b685f5-e45c-4c9c-a170-6d26a643341b-tls\") pod \"korrel8r-5cbcc89b8f-xnsgw\" (UID: \"07b685f5-e45c-4c9c-a170-6d26a643341b\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"
Oct 30 04:14:42.481123 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.481092    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"
Oct 30 04:14:42.525396 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.525369    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:42.773586 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.773551    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"]
Oct 30 04:14:42.777255 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:14:42.777230    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b685f5_e45c_4c9c_a170_6d26a643341b.slice/crio-dd3a120619dfe2e7d2c34ac6ab761466bf623fb1befe832f1d7f7e071d360604 WatchSource:0}: Error finding container dd3a120619dfe2e7d2c34ac6ab761466bf623fb1befe832f1d7f7e071d360604: Status 404 returned error can't find the container with id dd3a120619dfe2e7d2c34ac6ab761466bf623fb1befe832f1d7f7e071d360604
Oct 30 04:14:42.930825 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.930739    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw" event={"ID":"07b685f5-e45c-4c9c-a170-6d26a643341b","Type":"ContainerStarted","Data":"40e6478512dc6ecc17c3f3ad32a70cefacd4d27ea594fbe86a0f57d970aa451f"}
Oct 30 04:14:42.930825 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.930813    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw" event={"ID":"07b685f5-e45c-4c9c-a170-6d26a643341b","Type":"ContainerStarted","Data":"dd3a120619dfe2e7d2c34ac6ab761466bf623fb1befe832f1d7f7e071d360604"}
Oct 30 04:14:42.931223 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.930860    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"
Oct 30 04:14:42.933466 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.933432    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b" event={"ID":"ae264e2d-42ab-435c-a4f4-df406a2c9ee8","Type":"ContainerStarted","Data":"ba7260c0ff3a2ad81c4dc02bff000d1d7604ffbbc3faf34cfeaa4b96eeff7489"}
Oct 30 04:14:42.933590 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.933497    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b" event={"ID":"ae264e2d-42ab-435c-a4f4-df406a2c9ee8","Type":"ContainerStarted","Data":"507d5988471a128fb415479630790fe79ab1f9723c388bc3163c1fdf0b7e5521"}
Oct 30 04:14:42.933590 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.933531    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b" event={"ID":"ae264e2d-42ab-435c-a4f4-df406a2c9ee8","Type":"ContainerStarted","Data":"541e18575ce5e324f0209e5899d83c1cec1e3d13e607f747f949985949b7a0ab"}
Oct 30 04:14:42.935755 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.935723    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n" event={"ID":"02b485e3-c53a-4126-8854-1639da65cb05","Type":"ContainerStarted","Data":"d6eecf6272364353910c06209edf2df40f6e9c9a50de77b18b7cff6ddd4ea8e7"}
Oct 30 04:14:42.935866 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.935779    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n" event={"ID":"02b485e3-c53a-4126-8854-1639da65cb05","Type":"ContainerStarted","Data":"2ac5d83507c1643358d21712e0e69eafffca5dc3f7b531bc121fe4bc71c1f7e4"}
Oct 30 04:14:42.936297 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.936275    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:14:42.938546 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.938524    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq" event={"ID":"73fbee4b-fa9b-47e7-b350-427e832dd16c","Type":"ContainerStarted","Data":"0e5a8388796004afe15d0d51b823e6f167c3da65a710641f734755372c5b9513"}
Oct 30 04:14:42.938632 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.938569    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq" event={"ID":"73fbee4b-fa9b-47e7-b350-427e832dd16c","Type":"ContainerStarted","Data":"c7a92b416526254dcbf2b4ea07fe7fcc0d734c12c188e196d963aeda9c92e24e"}
Oct 30 04:14:42.938632 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.938600    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq" event={"ID":"73fbee4b-fa9b-47e7-b350-427e832dd16c","Type":"ContainerStarted","Data":"9d9405660fda43097585010c84f03b26cfcad339ffd0ca522b6f09846034f9e0"}
Oct 30 04:14:42.944607 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.944587    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:14:42.947985 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.947968    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:42.973777 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.973748    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:14:42.979523 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.979488    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:14:42.980487 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.980462    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:42.980593 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:42.980473    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw" podStartSLOduration=1.980457043 podStartE2EDuration="1.980457043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:42.979540461 +0000 UTC m=+5563.628588879" watchObservedRunningTime="2026-10-30 04:14:42.980457043 +0000 UTC m=+5563.629505463"
Oct 30 04:14:43.023864 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.023816    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b" podStartSLOduration=2.023802371 podStartE2EDuration="2.023802371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:43.023291356 +0000 UTC m=+5563.672339798" watchObservedRunningTime="2026-10-30 04:14:43.023802371 +0000 UTC m=+5563.672850791"
Oct 30 04:14:43.037747 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.037717    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:14:43.038008 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.037876    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:14:43.038387 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.038362    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dffss\""
Oct 30 04:14:43.038488 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.038466    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:14:43.038549 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.038527    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:14:43.065717 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.065689    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:43.065887 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.065823    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:43.065981 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.065884    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-424wb\" (UniqueName: \"kubernetes.io/projected/66922329-3872-4cd5-9783-a032a7d34d2b-kube-api-access-424wb\") pod \"data-science-collector-collector-1\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:43.066207 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.066168    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/66922329-3872-4cd5-9783-a032a7d34d2b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:43.066414 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.066391    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:43.066699 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.066662    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:43.102119 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.102089    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:43.108677 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.108634    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n" podStartSLOduration=2.108621657 podStartE2EDuration="2.108621657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:43.108050633 +0000 UTC m=+5563.757099056" watchObservedRunningTime="2026-10-30 04:14:43.108621657 +0000 UTC m=+5563.757670119"
Oct 30 04:14:43.167707 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.167671    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:43.168167 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.167765    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:43.168167 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.167793    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-424wb\" (UniqueName: \"kubernetes.io/projected/66922329-3872-4cd5-9783-a032a7d34d2b-kube-api-access-424wb\") pod \"data-science-collector-collector-1\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:43.168167 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.167830    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/66922329-3872-4cd5-9783-a032a7d34d2b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:43.168167 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.167862    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:43.168167 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.167894    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:43.168937 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.168856    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/66922329-3872-4cd5-9783-a032a7d34d2b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:43.171083 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.171053    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:43.171912 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.171888    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:43.175630 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.175580    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:43.176878 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.176856    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:43.191390 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.191333    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-424wb\" (UniqueName: \"kubernetes.io/projected/66922329-3872-4cd5-9783-a032a7d34d2b-kube-api-access-424wb\") pod \"data-science-collector-collector-1\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:43.214844 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.214819    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:43.277040 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.277015    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:43.569169 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.569116    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq" podStartSLOduration=2.569094662 podStartE2EDuration="2.569094662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:43.152485808 +0000 UTC m=+5563.801534230" watchObservedRunningTime="2026-10-30 04:14:43.569094662 +0000 UTC m=+5564.218143084"
Oct 30 04:14:43.569412 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.569394    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:14:43.570684 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:14:43.570659    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66922329_3872_4cd5_9783_a032a7d34d2b.slice/crio-d6169079ab3050d54a323f20bc638cebb2cb483b336960ced11a25ebe9b48e27 WatchSource:0}: Error finding container d6169079ab3050d54a323f20bc638cebb2cb483b336960ced11a25ebe9b48e27: Status 404 returned error can't find the container with id d6169079ab3050d54a323f20bc638cebb2cb483b336960ced11a25ebe9b48e27
Oct 30 04:14:43.572804 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.572771    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:14:43.572902 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.572842    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:14:43.946302 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.946206    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"66922329-3872-4cd5-9783-a032a7d34d2b","Type":"ContainerStarted","Data":"331fbfe1947ff50d313d6a27fe4b514e0e0040a60c5341d6b0127adb956ffb34"}
Oct 30 04:14:43.946302 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.946264    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"66922329-3872-4cd5-9783-a032a7d34d2b","Type":"ContainerStarted","Data":"d6169079ab3050d54a323f20bc638cebb2cb483b336960ced11a25ebe9b48e27"}
Oct 30 04:14:43.988841 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:43.988795    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.9887818940000002 podStartE2EDuration="1.988781894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:43.987340526 +0000 UTC m=+5564.636388948" watchObservedRunningTime="2026-10-30 04:14:43.988781894 +0000 UTC m=+5564.637830314"
Oct 30 04:14:45.790095 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:45.790063    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:46.024336 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.024305    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:46.137331 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.137296    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:46.374728 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.374695    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:14:46.378520 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.378498    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.405440 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.405385    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:46.411269 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.411244    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:14:46.411376 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.411362    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\""
Oct 30 04:14:46.411446 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.411405    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\""
Oct 30 04:14:46.411446 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.411433    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\""
Oct 30 04:14:46.411530 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.411492    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-2brw5\""
Oct 30 04:14:46.412692 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.412673    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:14:46.497145 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.497114    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.497330 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.497161    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c68gp\" (UniqueName: \"kubernetes.io/projected/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-kube-api-access-c68gp\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.497330 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.497253    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.497330 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.497290    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.497330 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.497318    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.497577 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.497410    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.497577 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.497455    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.519334 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.519307    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:46.578100 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.578074    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:14:46.583552 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.583525    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.598039 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.598013    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.598221 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.598100    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.598221 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.598161    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.598221 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.598208    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c68gp\" (UniqueName: \"kubernetes.io/projected/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-kube-api-access-c68gp\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.598505 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.598236    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.598505 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.598268    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.598505 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.598317    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.598676 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.598518    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.601532 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.601506    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.601532 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.601523    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.601681 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.601585    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.601681 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.601607    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.602162 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.602138    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.605450 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.605433    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 04:14:46.613746 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.613044    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:14:46.619214 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.619161    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c68gp\" (UniqueName: \"kubernetes.io/projected/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-kube-api-access-c68gp\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.662694 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.662641    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 04:14:46.662881 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.662867    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 04:14:46.662946 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.662907    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 04:14:46.696568 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.696544    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:46.698850 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.698829    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.698940 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.698868    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.698940 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.698922    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.699054 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.699036    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.699094 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.699083    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.699189 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.699164    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.699273 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.699253    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.699325 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.699301    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.699365 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.699354    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.699402 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.699378    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.699682 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.699403    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsg7j\" (UniqueName: \"kubernetes.io/projected/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-kube-api-access-bsg7j\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.699682 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.699432    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.701221 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.701197    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 04:14:46.701221 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.701207    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:14:46.701349 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.701207    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 04:14:46.701349 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.701267    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:14:46.701349 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.701270    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-9zb5b\""
Oct 30 04:14:46.800756 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.800726    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.801158 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.800770    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.801158 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.800798    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.801158 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.800845    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.801158 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.800874    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.801158 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.800996    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsg7j\" (UniqueName: \"kubernetes.io/projected/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-kube-api-access-bsg7j\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.801158 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.801052    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.801158 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.801090    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.801158 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.801128    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.801652 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.801201    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.801652 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.801240    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.801652 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.801277    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.801652 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.801638    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.801861 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.801736    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.802009 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.801979    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.804438 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.804408    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.804553 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.804492    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.804908 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.804883    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.805055 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.805032    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.805342 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.805323    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.805623 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.805602    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.805708 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.805621    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.805840 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.805822    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:46.824295 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:46.824276    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsg7j\" (UniqueName: \"kubernetes.io/projected/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-kube-api-access-bsg7j\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:47.001499 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:47.001479    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:14:47.002796 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:14:47.002774    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4c28df_ac78_4d9e_99cb_26b725bc9c79.slice/crio-a3ffd3cbed07d0f723705fb497cade1d0e6a383e4e93670b43e377fc3a28764d WatchSource:0}: Error finding container a3ffd3cbed07d0f723705fb497cade1d0e6a383e4e93670b43e377fc3a28764d: Status 404 returned error can't find the container with id a3ffd3cbed07d0f723705fb497cade1d0e6a383e4e93670b43e377fc3a28764d
Oct 30 04:14:47.968280 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:47.968239    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2c4c28df-ac78-4d9e-99cb-26b725bc9c79","Type":"ContainerStarted","Data":"67ea680d183ed7f574586926e11e53c4a9b9e0c0d5dafde85eaa1ba3899b5ab1"}
Oct 30 04:14:47.968650 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:47.968323    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2c4c28df-ac78-4d9e-99cb-26b725bc9c79","Type":"ContainerStarted","Data":"a3ffd3cbed07d0f723705fb497cade1d0e6a383e4e93670b43e377fc3a28764d"}
Oct 30 04:14:49.954068 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:49.954042    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:14:50.984221 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:50.984159    4960 generic.go:356] "Generic (PLEG): container finished" podID="2c4c28df-ac78-4d9e-99cb-26b725bc9c79" containerID="67ea680d183ed7f574586926e11e53c4a9b9e0c0d5dafde85eaa1ba3899b5ab1" exitCode=0
Oct 30 04:14:50.984597 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:50.984231    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2c4c28df-ac78-4d9e-99cb-26b725bc9c79","Type":"ContainerDied","Data":"67ea680d183ed7f574586926e11e53c4a9b9e0c0d5dafde85eaa1ba3899b5ab1"}
Oct 30 04:14:51.639576 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:51.639548    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:51.670755 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:51.670720    4960 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:51.740325 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:51.740303    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:51.740448 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:51.740427    4960 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:51.755874 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:51.755813    4960 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") DevicePath \"csi-2479454746a6578572f425530fc81bfe97655fb31272a4af03192f10a097ebe5\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:51.858837 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:51.858814    4960 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6b615f98501c6b5529f28e27a4cb6d663be03751269539a798523d989e4446fc/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:51.886958 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:51.886933    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:51.991596 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:51.991567    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2c4c28df-ac78-4d9e-99cb-26b725bc9c79","Type":"ContainerStarted","Data":"8f3c52c589bead205a0a30b44746f898cba39f2713babafacb96266a953cdb9c"}
Oct 30 04:14:51.991887 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:51.991627    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2c4c28df-ac78-4d9e-99cb-26b725bc9c79","Type":"ContainerStarted","Data":"a7e939893a308f5f0c7f015d57be414b45e8a20ccad4f10857138cbc8a51d5c4"}
Oct 30 04:14:51.991887 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:51.991808    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:52.005697 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:52.005674    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:52.041301 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:52.041254    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=2.554623761 podStartE2EDuration="6.041241094s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.486617333s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:52.039129375 +0000 UTC m=+5572.688177796" watchObservedRunningTime="2026-10-30 04:14:52.041241094 +0000 UTC m=+5572.690289551"
Oct 30 04:14:52.326784 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:52.326750    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:14:52.337691 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:14:52.337657    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c36ec9_29c3_4b59_a82e_7da81e0e6a08.slice/crio-0743f414becbcdbd0a17044789e5581c893fad9c33205969ed2dddea9b6f6a45 WatchSource:0}: Error finding container 0743f414becbcdbd0a17044789e5581c893fad9c33205969ed2dddea9b6f6a45: Status 404 returned error can't find the container with id 0743f414becbcdbd0a17044789e5581c893fad9c33205969ed2dddea9b6f6a45
Oct 30 04:14:52.998048 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:52.997979    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"28c36ec9-29c3-4b59-a82e-7da81e0e6a08","Type":"ContainerStarted","Data":"1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689"}
Oct 30 04:14:52.998381 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:52.998054    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"28c36ec9-29c3-4b59-a82e-7da81e0e6a08","Type":"ContainerStarted","Data":"0743f414becbcdbd0a17044789e5581c893fad9c33205969ed2dddea9b6f6a45"}
Oct 30 04:14:53.948806 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:53.948780    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"
Oct 30 04:14:53.953992 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:14:53.953961    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:14:57.018947 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:57.018902    4960 generic.go:356] "Generic (PLEG): container finished" podID="28c36ec9-29c3-4b59-a82e-7da81e0e6a08" containerID="1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689" exitCode=0
Oct 30 04:14:57.019409 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:57.018934    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"28c36ec9-29c3-4b59-a82e-7da81e0e6a08","Type":"ContainerDied","Data":"1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689"}
Oct 30 04:14:57.999871 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:57.999840    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:14:58.026601 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:58.026575    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"28c36ec9-29c3-4b59-a82e-7da81e0e6a08","Type":"ContainerStarted","Data":"682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0"}
Oct 30 04:14:58.026934 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:58.026641    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"28c36ec9-29c3-4b59-a82e-7da81e0e6a08","Type":"ContainerStarted","Data":"bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd"}
Oct 30 04:14:58.026934 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:58.026669    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"28c36ec9-29c3-4b59-a82e-7da81e0e6a08","Type":"ContainerStarted","Data":"a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451"}
Oct 30 04:14:58.093760 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:14:58.093703    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=8.610022558 podStartE2EDuration="12.093684817s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.483662259s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:58.092234602 +0000 UTC m=+5578.741283024" watchObservedRunningTime="2026-10-30 04:14:58.093684817 +0000 UTC m=+5578.742733238"
Oct 30 04:15:00.179276 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:00.179237    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:01.329224 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:01.329160    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:01.526202 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:01.526141    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:01.640371 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:01.640297    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:02.006045 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:02.005970    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:03.028761 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.028726    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:15:03.029144 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.028905    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="66922329-3872-4cd5-9783-a032a7d34d2b" containerName="otc-container" containerID="cri-o://331fbfe1947ff50d313d6a27fe4b514e0e0040a60c5341d6b0127adb956ffb34" gracePeriod=30
Oct 30 04:15:03.164777 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.164743    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:03.277191 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.277140    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:03.291305 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.291255    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:03.433908 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.433879    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-monitoring-tls-certs\") pod \"66922329-3872-4cd5-9783-a032a7d34d2b\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") "
Oct 30 04:15:03.434092 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.433924    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66922329-3872-4cd5-9783-a032a7d34d2b-kube-api-access-424wb\" (UniqueName: \"kubernetes.io/projected/66922329-3872-4cd5-9783-a032a7d34d2b-kube-api-access-424wb\") pod \"66922329-3872-4cd5-9783-a032a7d34d2b\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") "
Oct 30 04:15:03.434092 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.433994    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-otlp-tls-certs\") pod \"66922329-3872-4cd5-9783-a032a7d34d2b\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") "
Oct 30 04:15:03.434092 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.434025    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-tls-certs\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-tls-certs\") pod \"66922329-3872-4cd5-9783-a032a7d34d2b\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") "
Oct 30 04:15:03.434092 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.434081    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66922329-3872-4cd5-9783-a032a7d34d2b-otc-internal\" (UniqueName: \"kubernetes.io/configmap/66922329-3872-4cd5-9783-a032a7d34d2b-otc-internal\") pod \"66922329-3872-4cd5-9783-a032a7d34d2b\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") "
Oct 30 04:15:03.434388 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.434134    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-data-science-collector-ta-client-cert\") pod \"66922329-3872-4cd5-9783-a032a7d34d2b\" (UID: \"66922329-3872-4cd5-9783-a032a7d34d2b\") "
Oct 30 04:15:03.434548 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.434516    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66922329-3872-4cd5-9783-a032a7d34d2b-otc-internal" pod "66922329-3872-4cd5-9783-a032a7d34d2b" (UID: "66922329-3872-4cd5-9783-a032a7d34d2b"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:03.436479 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.436447    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-otlp-tls-certs" pod "66922329-3872-4cd5-9783-a032a7d34d2b" (UID: "66922329-3872-4cd5-9783-a032a7d34d2b"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:03.436585 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.436504    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-tls-certs" pod "66922329-3872-4cd5-9783-a032a7d34d2b" (UID: "66922329-3872-4cd5-9783-a032a7d34d2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:03.436632 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.436572    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-data-science-collector-ta-client-cert" pod "66922329-3872-4cd5-9783-a032a7d34d2b" (UID: "66922329-3872-4cd5-9783-a032a7d34d2b"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:03.436739 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.436721    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-monitoring-tls-certs" pod "66922329-3872-4cd5-9783-a032a7d34d2b" (UID: "66922329-3872-4cd5-9783-a032a7d34d2b"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:03.436898 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.436881    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66922329-3872-4cd5-9783-a032a7d34d2b-kube-api-access-424wb" pod "66922329-3872-4cd5-9783-a032a7d34d2b" (UID: "66922329-3872-4cd5-9783-a032a7d34d2b"). InnerVolumeSpecName "kube-api-access-424wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:03.534987 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.534963    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:03.535114 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.534989    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:03.535114 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.535000    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-424wb\" (UniqueName: \"kubernetes.io/projected/66922329-3872-4cd5-9783-a032a7d34d2b-kube-api-access-424wb\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:03.535114 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.535011    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:03.535114 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.535020    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66922329-3872-4cd5-9783-a032a7d34d2b-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:03.535114 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:03.535028    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/66922329-3872-4cd5-9783-a032a7d34d2b-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:04.056488 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:04.056456    4960 generic.go:356] "Generic (PLEG): container finished" podID="66922329-3872-4cd5-9783-a032a7d34d2b" containerID="331fbfe1947ff50d313d6a27fe4b514e0e0040a60c5341d6b0127adb956ffb34" exitCode=0
Oct 30 04:15:04.056977 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:04.056504    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:04.056977 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:04.056546    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"66922329-3872-4cd5-9783-a032a7d34d2b","Type":"ContainerDied","Data":"331fbfe1947ff50d313d6a27fe4b514e0e0040a60c5341d6b0127adb956ffb34"}
Oct 30 04:15:04.056977 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:04.056610    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"66922329-3872-4cd5-9783-a032a7d34d2b","Type":"ContainerDied","Data":"d6169079ab3050d54a323f20bc638cebb2cb483b336960ced11a25ebe9b48e27"}
Oct 30 04:15:04.056977 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:04.056644    4960 scope.go:122] "RemoveContainer" containerID="331fbfe1947ff50d313d6a27fe4b514e0e0040a60c5341d6b0127adb956ffb34"
Oct 30 04:15:04.065322 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:04.065304    4960 scope.go:122] "RemoveContainer" containerID="331fbfe1947ff50d313d6a27fe4b514e0e0040a60c5341d6b0127adb956ffb34"
Oct 30 04:15:04.065596 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:15:04.065574    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"331fbfe1947ff50d313d6a27fe4b514e0e0040a60c5341d6b0127adb956ffb34\": container with ID starting with 331fbfe1947ff50d313d6a27fe4b514e0e0040a60c5341d6b0127adb956ffb34 not found: ID does not exist" containerID="331fbfe1947ff50d313d6a27fe4b514e0e0040a60c5341d6b0127adb956ffb34"
Oct 30 04:15:04.065639 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:04.065608    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"331fbfe1947ff50d313d6a27fe4b514e0e0040a60c5341d6b0127adb956ffb34"} err="failed to get container status \"331fbfe1947ff50d313d6a27fe4b514e0e0040a60c5341d6b0127adb956ffb34\": rpc error: code = NotFound desc = could not find container \"331fbfe1947ff50d313d6a27fe4b514e0e0040a60c5341d6b0127adb956ffb34\": container with ID starting with 331fbfe1947ff50d313d6a27fe4b514e0e0040a60c5341d6b0127adb956ffb34 not found: ID does not exist"
Oct 30 04:15:04.110222 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:04.110193    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:15:04.123900 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:04.123873    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:15:05.504675 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:05.504641    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66922329-3872-4cd5-9783-a032a7d34d2b" path="/var/lib/kubelet/pods/66922329-3872-4cd5-9783-a032a7d34d2b/volumes"
Oct 30 04:15:07.006794 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:07.006761    4960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:07.011933 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:07.011912    4960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:07.076797 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:07.076774    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:12.594268 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:12.594232    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:12.699725 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:12.699697    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:12.804349 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:12.804318    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:12.918907 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:12.918827    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:14.414535 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:14.414502    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:14.529447 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:14.529419    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:14.630276 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:14.630244    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:16.300490 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:16.300452    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:18.526638 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.526608    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"]
Oct 30 04:15:18.527035 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.526806    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n" podUID="02b485e3-c53a-4126-8854-1639da65cb05" containerName="kube-rbac-proxy" containerID="cri-o://d6eecf6272364353910c06209edf2df40f6e9c9a50de77b18b7cff6ddd4ea8e7" gracePeriod=30
Oct 30 04:15:18.540025 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.539998    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"]
Oct 30 04:15:18.540276 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.540252    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq" podUID="73fbee4b-fa9b-47e7-b350-427e832dd16c" containerName="prom-label-proxy" containerID="cri-o://0e5a8388796004afe15d0d51b823e6f167c3da65a710641f734755372c5b9513" gracePeriod=30
Oct 30 04:15:18.540322 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.540281    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq" podUID="73fbee4b-fa9b-47e7-b350-427e832dd16c" containerName="kube-rbac-proxy" containerID="cri-o://c7a92b416526254dcbf2b4ea07fe7fcc0d734c12c188e196d963aeda9c92e24e" gracePeriod=30
Oct 30 04:15:18.554217 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.554193    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b"]
Oct 30 04:15:18.554429 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.554408    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b" podUID="ae264e2d-42ab-435c-a4f4-df406a2c9ee8" containerName="kube-rbac-proxy" containerID="cri-o://507d5988471a128fb415479630790fe79ab1f9723c388bc3163c1fdf0b7e5521" gracePeriod=30
Oct 30 04:15:18.554514 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.554415    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b" podUID="ae264e2d-42ab-435c-a4f4-df406a2c9ee8" containerName="prom-label-proxy" containerID="cri-o://ba7260c0ff3a2ad81c4dc02bff000d1d7604ffbbc3faf34cfeaa4b96eeff7489" gracePeriod=30
Oct 30 04:15:18.579024 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.578993    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"]
Oct 30 04:15:18.580208 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.579301    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw" podUID="07b685f5-e45c-4c9c-a170-6d26a643341b" containerName="korrel8r" containerID="cri-o://40e6478512dc6ecc17c3f3ad32a70cefacd4d27ea594fbe86a0f57d970aa451f" gracePeriod=30
Oct 30 04:15:18.836225 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.836196    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:15:18.874425 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.874394    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"
Oct 30 04:15:18.897142 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.896384    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b"
Oct 30 04:15:18.942104 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.942075    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"
Oct 30 04:15:18.961325 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.960354    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-tls\" (UniqueName: \"kubernetes.io/secret/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-tls\") pod \"ae264e2d-42ab-435c-a4f4-df406a2c9ee8\" (UID: \"ae264e2d-42ab-435c-a4f4-df406a2c9ee8\") "
Oct 30 04:15:18.961325 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.960413    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-kube-api-access-cr7w9\" (UniqueName: \"kubernetes.io/projected/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-kube-api-access-cr7w9\") pod \"ae264e2d-42ab-435c-a4f4-df406a2c9ee8\" (UID: \"ae264e2d-42ab-435c-a4f4-df406a2c9ee8\") "
Oct 30 04:15:18.961325 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.960448    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-kube-rbac-proxy-config\") pod \"02b485e3-c53a-4126-8854-1639da65cb05\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") "
Oct 30 04:15:18.961325 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.960510    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-config\" (UniqueName: \"kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-config\") pod \"73fbee4b-fa9b-47e7-b350-427e832dd16c\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") "
Oct 30 04:15:18.961325 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.960936    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-config" pod "73fbee4b-fa9b-47e7-b350-427e832dd16c" (UID: "73fbee4b-fa9b-47e7-b350-427e832dd16c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:18.961325 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.961014    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-prometheus-tls-ca\") pod \"73fbee4b-fa9b-47e7-b350-427e832dd16c\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") "
Oct 30 04:15:18.961325 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.961107    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73fbee4b-fa9b-47e7-b350-427e832dd16c-tls\" (UniqueName: \"kubernetes.io/secret/73fbee4b-fa9b-47e7-b350-427e832dd16c-tls\") pod \"73fbee4b-fa9b-47e7-b350-427e832dd16c\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") "
Oct 30 04:15:18.961325 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.961252    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-tls\" (UniqueName: \"kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-tls\") pod \"02b485e3-c53a-4126-8854-1639da65cb05\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") "
Oct 30 04:15:18.961325 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.961314    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-prometheus-tls-ca" pod "73fbee4b-fa9b-47e7-b350-427e832dd16c" (UID: "73fbee4b-fa9b-47e7-b350-427e832dd16c"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:18.962099 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.961336    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-config\" (UniqueName: \"kubernetes.io/configmap/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-config\") pod \"ae264e2d-42ab-435c-a4f4-df406a2c9ee8\" (UID: \"ae264e2d-42ab-435c-a4f4-df406a2c9ee8\") "
Oct 30 04:15:18.962099 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.961377    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-client-ca\" (UniqueName: \"kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-client-ca\") pod \"73fbee4b-fa9b-47e7-b350-427e832dd16c\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") "
Oct 30 04:15:18.962099 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.961413    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73fbee4b-fa9b-47e7-b350-427e832dd16c-kube-api-access-c6wsc\" (UniqueName: \"kubernetes.io/projected/73fbee4b-fa9b-47e7-b350-427e832dd16c-kube-api-access-c6wsc\") pod \"73fbee4b-fa9b-47e7-b350-427e832dd16c\" (UID: \"73fbee4b-fa9b-47e7-b350-427e832dd16c\") "
Oct 30 04:15:18.962099 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.961450    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b485e3-c53a-4126-8854-1639da65cb05-kube-api-access-p4gff\" (UniqueName: \"kubernetes.io/projected/02b485e3-c53a-4126-8854-1639da65cb05-kube-api-access-p4gff\") pod \"02b485e3-c53a-4126-8854-1639da65cb05\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") "
Oct 30 04:15:18.962099 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.961480    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02b485e3-c53a-4126-8854-1639da65cb05-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/02b485e3-c53a-4126-8854-1639da65cb05-prometheus-ca\") pod \"02b485e3-c53a-4126-8854-1639da65cb05\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") "
Oct 30 04:15:18.962099 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.961510    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-prometheus-client-cert\") pod \"02b485e3-c53a-4126-8854-1639da65cb05\" (UID: \"02b485e3-c53a-4126-8854-1639da65cb05\") "
Oct 30 04:15:18.962442 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.962406    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:18.962442 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.962431    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-prometheus-tls-ca\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:18.962553 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.962529    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-config" pod "ae264e2d-42ab-435c-a4f4-df406a2c9ee8" (UID: "ae264e2d-42ab-435c-a4f4-df406a2c9ee8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:18.962921 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.962894    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02b485e3-c53a-4126-8854-1639da65cb05-prometheus-ca" pod "02b485e3-c53a-4126-8854-1639da65cb05" (UID: "02b485e3-c53a-4126-8854-1639da65cb05"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:18.963306 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.963270    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-client-ca" pod "73fbee4b-fa9b-47e7-b350-427e832dd16c" (UID: "73fbee4b-fa9b-47e7-b350-427e832dd16c"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:18.965776 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.965745    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73fbee4b-fa9b-47e7-b350-427e832dd16c-kube-api-access-c6wsc" pod "73fbee4b-fa9b-47e7-b350-427e832dd16c" (UID: "73fbee4b-fa9b-47e7-b350-427e832dd16c"). InnerVolumeSpecName "kube-api-access-c6wsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:18.966418 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.966392    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-prometheus-client-cert" pod "02b485e3-c53a-4126-8854-1639da65cb05" (UID: "02b485e3-c53a-4126-8854-1639da65cb05"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:18.966637 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.966619    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-tls" pod "02b485e3-c53a-4126-8854-1639da65cb05" (UID: "02b485e3-c53a-4126-8854-1639da65cb05"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:18.966750 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.966731    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-kube-api-access-cr7w9" pod "ae264e2d-42ab-435c-a4f4-df406a2c9ee8" (UID: "ae264e2d-42ab-435c-a4f4-df406a2c9ee8"). InnerVolumeSpecName "kube-api-access-cr7w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:18.967037 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.966765    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b485e3-c53a-4126-8854-1639da65cb05-kube-api-access-p4gff" pod "02b485e3-c53a-4126-8854-1639da65cb05" (UID: "02b485e3-c53a-4126-8854-1639da65cb05"). InnerVolumeSpecName "kube-api-access-p4gff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:18.967037 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.966876    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-kube-rbac-proxy-config" pod "02b485e3-c53a-4126-8854-1639da65cb05" (UID: "02b485e3-c53a-4126-8854-1639da65cb05"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:18.967037 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.966991    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-tls" pod "ae264e2d-42ab-435c-a4f4-df406a2c9ee8" (UID: "ae264e2d-42ab-435c-a4f4-df406a2c9ee8"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:18.968496 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.968470    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73fbee4b-fa9b-47e7-b350-427e832dd16c-tls" pod "73fbee4b-fa9b-47e7-b350-427e832dd16c" (UID: "73fbee4b-fa9b-47e7-b350-427e832dd16c"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:18.999561 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:18.999513    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:19.038038 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.037956    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:15:19.038358 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.038327    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="2c4c28df-ac78-4d9e-99cb-26b725bc9c79" containerName="alertmanager" containerID="cri-o://a7e939893a308f5f0c7f015d57be414b45e8a20ccad4f10857138cbc8a51d5c4" gracePeriod=120
Oct 30 04:15:19.038735 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.038698    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="2c4c28df-ac78-4d9e-99cb-26b725bc9c79" containerName="config-reloader" containerID="cri-o://8f3c52c589bead205a0a30b44746f898cba39f2713babafacb96266a953cdb9c" gracePeriod=120
Oct 30 04:15:19.040080 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.040057    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:15:19.040365 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.040342    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="28c36ec9-29c3-4b59-a82e-7da81e0e6a08" containerName="prometheus" containerID="cri-o://a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451" gracePeriod=600
Oct 30 04:15:19.041289 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.040367    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="28c36ec9-29c3-4b59-a82e-7da81e0e6a08" containerName="thanos-sidecar" containerID="cri-o://682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0" gracePeriod=600
Oct 30 04:15:19.041289 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.040379    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="28c36ec9-29c3-4b59-a82e-7da81e0e6a08" containerName="config-reloader" containerID="cri-o://bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd" gracePeriod=600
Oct 30 04:15:19.063254 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.063228    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07b685f5-e45c-4c9c-a170-6d26a643341b-tls\" (UniqueName: \"kubernetes.io/secret/07b685f5-e45c-4c9c-a170-6d26a643341b-tls\") pod \"07b685f5-e45c-4c9c-a170-6d26a643341b\" (UID: \"07b685f5-e45c-4c9c-a170-6d26a643341b\") "
Oct 30 04:15:19.063360 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.063292    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b685f5-e45c-4c9c-a170-6d26a643341b-kube-api-access-tf8kw\" (UniqueName: \"kubernetes.io/projected/07b685f5-e45c-4c9c-a170-6d26a643341b-kube-api-access-tf8kw\") pod \"07b685f5-e45c-4c9c-a170-6d26a643341b\" (UID: \"07b685f5-e45c-4c9c-a170-6d26a643341b\") "
Oct 30 04:15:19.063360 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.063319    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07b685f5-e45c-4c9c-a170-6d26a643341b-config\" (UniqueName: \"kubernetes.io/configmap/07b685f5-e45c-4c9c-a170-6d26a643341b-config\") pod \"07b685f5-e45c-4c9c-a170-6d26a643341b\" (UID: \"07b685f5-e45c-4c9c-a170-6d26a643341b\") "
Oct 30 04:15:19.063573 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.063555    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/73fbee4b-fa9b-47e7-b350-427e832dd16c-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.063648 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.063578    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.063648 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.063591    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.063648 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.063603    4960 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/73fbee4b-fa9b-47e7-b350-427e832dd16c-client-ca\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.063648 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.063614    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6wsc\" (UniqueName: \"kubernetes.io/projected/73fbee4b-fa9b-47e7-b350-427e832dd16c-kube-api-access-c6wsc\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.063648 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.063629    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4gff\" (UniqueName: \"kubernetes.io/projected/02b485e3-c53a-4126-8854-1639da65cb05-kube-api-access-p4gff\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.063648 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.063645    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/02b485e3-c53a-4126-8854-1639da65cb05-prometheus-ca\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.063868 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.063660    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-prometheus-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.063868 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.063673    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.063868 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.063686    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr7w9\" (UniqueName: \"kubernetes.io/projected/ae264e2d-42ab-435c-a4f4-df406a2c9ee8-kube-api-access-cr7w9\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.063868 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.063699    4960 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02b485e3-c53a-4126-8854-1639da65cb05-kube-rbac-proxy-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.063868 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.063759    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b685f5-e45c-4c9c-a170-6d26a643341b-config" pod "07b685f5-e45c-4c9c-a170-6d26a643341b" (UID: "07b685f5-e45c-4c9c-a170-6d26a643341b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:19.065911 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.065884    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b685f5-e45c-4c9c-a170-6d26a643341b-kube-api-access-tf8kw" pod "07b685f5-e45c-4c9c-a170-6d26a643341b" (UID: "07b685f5-e45c-4c9c-a170-6d26a643341b"). InnerVolumeSpecName "kube-api-access-tf8kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:19.066279 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.066258    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07b685f5-e45c-4c9c-a170-6d26a643341b-tls" pod "07b685f5-e45c-4c9c-a170-6d26a643341b" (UID: "07b685f5-e45c-4c9c-a170-6d26a643341b"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:19.127111 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.127084    4960 generic.go:356] "Generic (PLEG): container finished" podID="73fbee4b-fa9b-47e7-b350-427e832dd16c" containerID="0e5a8388796004afe15d0d51b823e6f167c3da65a710641f734755372c5b9513" exitCode=0
Oct 30 04:15:19.127111 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.127105    4960 generic.go:356] "Generic (PLEG): container finished" podID="73fbee4b-fa9b-47e7-b350-427e832dd16c" containerID="c7a92b416526254dcbf2b4ea07fe7fcc0d734c12c188e196d963aeda9c92e24e" exitCode=0
Oct 30 04:15:19.127345 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.127128    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"
Oct 30 04:15:19.127345 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.127204    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq" event={"ID":"73fbee4b-fa9b-47e7-b350-427e832dd16c","Type":"ContainerDied","Data":"0e5a8388796004afe15d0d51b823e6f167c3da65a710641f734755372c5b9513"}
Oct 30 04:15:19.127345 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.127276    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq" event={"ID":"73fbee4b-fa9b-47e7-b350-427e832dd16c","Type":"ContainerDied","Data":"c7a92b416526254dcbf2b4ea07fe7fcc0d734c12c188e196d963aeda9c92e24e"}
Oct 30 04:15:19.127345 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.127312    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq" event={"ID":"73fbee4b-fa9b-47e7-b350-427e832dd16c","Type":"ContainerDied","Data":"9d9405660fda43097585010c84f03b26cfcad339ffd0ca522b6f09846034f9e0"}
Oct 30 04:15:19.127503 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.127345    4960 scope.go:122] "RemoveContainer" containerID="0e5a8388796004afe15d0d51b823e6f167c3da65a710641f734755372c5b9513"
Oct 30 04:15:19.128720 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.128694    4960 generic.go:356] "Generic (PLEG): container finished" podID="07b685f5-e45c-4c9c-a170-6d26a643341b" containerID="40e6478512dc6ecc17c3f3ad32a70cefacd4d27ea594fbe86a0f57d970aa451f" exitCode=0
Oct 30 04:15:19.128821 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.128790    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw" event={"ID":"07b685f5-e45c-4c9c-a170-6d26a643341b","Type":"ContainerDied","Data":"40e6478512dc6ecc17c3f3ad32a70cefacd4d27ea594fbe86a0f57d970aa451f"}
Oct 30 04:15:19.128895 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.128851    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw" event={"ID":"07b685f5-e45c-4c9c-a170-6d26a643341b","Type":"ContainerDied","Data":"dd3a120619dfe2e7d2c34ac6ab761466bf623fb1befe832f1d7f7e071d360604"}
Oct 30 04:15:19.128977 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.128960    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"
Oct 30 04:15:19.130826 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.130803    4960 generic.go:356] "Generic (PLEG): container finished" podID="ae264e2d-42ab-435c-a4f4-df406a2c9ee8" containerID="ba7260c0ff3a2ad81c4dc02bff000d1d7604ffbbc3faf34cfeaa4b96eeff7489" exitCode=0
Oct 30 04:15:19.130826 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.130823    4960 generic.go:356] "Generic (PLEG): container finished" podID="ae264e2d-42ab-435c-a4f4-df406a2c9ee8" containerID="507d5988471a128fb415479630790fe79ab1f9723c388bc3163c1fdf0b7e5521" exitCode=0
Oct 30 04:15:19.131015 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.130854    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b"
Oct 30 04:15:19.131015 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.130915    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b" event={"ID":"ae264e2d-42ab-435c-a4f4-df406a2c9ee8","Type":"ContainerDied","Data":"ba7260c0ff3a2ad81c4dc02bff000d1d7604ffbbc3faf34cfeaa4b96eeff7489"}
Oct 30 04:15:19.131015 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.130961    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b" event={"ID":"ae264e2d-42ab-435c-a4f4-df406a2c9ee8","Type":"ContainerDied","Data":"507d5988471a128fb415479630790fe79ab1f9723c388bc3163c1fdf0b7e5521"}
Oct 30 04:15:19.131015 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.130990    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b" event={"ID":"ae264e2d-42ab-435c-a4f4-df406a2c9ee8","Type":"ContainerDied","Data":"541e18575ce5e324f0209e5899d83c1cec1e3d13e607f747f949985949b7a0ab"}
Oct 30 04:15:19.132576 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.132554    4960 generic.go:356] "Generic (PLEG): container finished" podID="02b485e3-c53a-4126-8854-1639da65cb05" containerID="d6eecf6272364353910c06209edf2df40f6e9c9a50de77b18b7cff6ddd4ea8e7" exitCode=0
Oct 30 04:15:19.132699 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.132610    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n" event={"ID":"02b485e3-c53a-4126-8854-1639da65cb05","Type":"ContainerDied","Data":"d6eecf6272364353910c06209edf2df40f6e9c9a50de77b18b7cff6ddd4ea8e7"}
Oct 30 04:15:19.132699 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.132643    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n" event={"ID":"02b485e3-c53a-4126-8854-1639da65cb05","Type":"ContainerDied","Data":"2ac5d83507c1643358d21712e0e69eafffca5dc3f7b531bc121fe4bc71c1f7e4"}
Oct 30 04:15:19.132699 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.132689    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"
Oct 30 04:15:19.133212 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.133169    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:19.164979 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.164962    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/07b685f5-e45c-4c9c-a170-6d26a643341b-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.165063 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.164982    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf8kw\" (UniqueName: \"kubernetes.io/projected/07b685f5-e45c-4c9c-a170-6d26a643341b-kube-api-access-tf8kw\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.165063 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.164991    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07b685f5-e45c-4c9c-a170-6d26a643341b-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.199218 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.199166    4960 scope.go:122] "RemoveContainer" containerID="c7a92b416526254dcbf2b4ea07fe7fcc0d734c12c188e196d963aeda9c92e24e"
Oct 30 04:15:19.237743 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.237719    4960 scope.go:122] "RemoveContainer" containerID="0e5a8388796004afe15d0d51b823e6f167c3da65a710641f734755372c5b9513"
Oct 30 04:15:19.238137 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:15:19.238115    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5a8388796004afe15d0d51b823e6f167c3da65a710641f734755372c5b9513\": container with ID starting with 0e5a8388796004afe15d0d51b823e6f167c3da65a710641f734755372c5b9513 not found: ID does not exist" containerID="0e5a8388796004afe15d0d51b823e6f167c3da65a710641f734755372c5b9513"
Oct 30 04:15:19.238204 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.238150    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5a8388796004afe15d0d51b823e6f167c3da65a710641f734755372c5b9513"} err="failed to get container status \"0e5a8388796004afe15d0d51b823e6f167c3da65a710641f734755372c5b9513\": rpc error: code = NotFound desc = could not find container \"0e5a8388796004afe15d0d51b823e6f167c3da65a710641f734755372c5b9513\": container with ID starting with 0e5a8388796004afe15d0d51b823e6f167c3da65a710641f734755372c5b9513 not found: ID does not exist"
Oct 30 04:15:19.238204 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.238168    4960 scope.go:122] "RemoveContainer" containerID="c7a92b416526254dcbf2b4ea07fe7fcc0d734c12c188e196d963aeda9c92e24e"
Oct 30 04:15:19.238481 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:15:19.238458    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a92b416526254dcbf2b4ea07fe7fcc0d734c12c188e196d963aeda9c92e24e\": container with ID starting with c7a92b416526254dcbf2b4ea07fe7fcc0d734c12c188e196d963aeda9c92e24e not found: ID does not exist" containerID="c7a92b416526254dcbf2b4ea07fe7fcc0d734c12c188e196d963aeda9c92e24e"
Oct 30 04:15:19.238538 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.238490    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a92b416526254dcbf2b4ea07fe7fcc0d734c12c188e196d963aeda9c92e24e"} err="failed to get container status \"c7a92b416526254dcbf2b4ea07fe7fcc0d734c12c188e196d963aeda9c92e24e\": rpc error: code = NotFound desc = could not find container \"c7a92b416526254dcbf2b4ea07fe7fcc0d734c12c188e196d963aeda9c92e24e\": container with ID starting with c7a92b416526254dcbf2b4ea07fe7fcc0d734c12c188e196d963aeda9c92e24e not found: ID does not exist"
Oct 30 04:15:19.238538 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.238513    4960 scope.go:122] "RemoveContainer" containerID="0e5a8388796004afe15d0d51b823e6f167c3da65a710641f734755372c5b9513"
Oct 30 04:15:19.238852 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.238825    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5a8388796004afe15d0d51b823e6f167c3da65a710641f734755372c5b9513"} err="failed to get container status \"0e5a8388796004afe15d0d51b823e6f167c3da65a710641f734755372c5b9513\": rpc error: code = NotFound desc = could not find container \"0e5a8388796004afe15d0d51b823e6f167c3da65a710641f734755372c5b9513\": container with ID starting with 0e5a8388796004afe15d0d51b823e6f167c3da65a710641f734755372c5b9513 not found: ID does not exist"
Oct 30 04:15:19.238942 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.238855    4960 scope.go:122] "RemoveContainer" containerID="c7a92b416526254dcbf2b4ea07fe7fcc0d734c12c188e196d963aeda9c92e24e"
Oct 30 04:15:19.239141 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.239113    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a92b416526254dcbf2b4ea07fe7fcc0d734c12c188e196d963aeda9c92e24e"} err="failed to get container status \"c7a92b416526254dcbf2b4ea07fe7fcc0d734c12c188e196d963aeda9c92e24e\": rpc error: code = NotFound desc = could not find container \"c7a92b416526254dcbf2b4ea07fe7fcc0d734c12c188e196d963aeda9c92e24e\": container with ID starting with c7a92b416526254dcbf2b4ea07fe7fcc0d734c12c188e196d963aeda9c92e24e not found: ID does not exist"
Oct 30 04:15:19.239220 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.239143    4960 scope.go:122] "RemoveContainer" containerID="40e6478512dc6ecc17c3f3ad32a70cefacd4d27ea594fbe86a0f57d970aa451f"
Oct 30 04:15:19.244622 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.244576    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"]
Oct 30 04:15:19.245938 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.245917    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-c822n"]
Oct 30 04:15:19.249962 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.249942    4960 scope.go:122] "RemoveContainer" containerID="40e6478512dc6ecc17c3f3ad32a70cefacd4d27ea594fbe86a0f57d970aa451f"
Oct 30 04:15:19.250251 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:15:19.250230    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e6478512dc6ecc17c3f3ad32a70cefacd4d27ea594fbe86a0f57d970aa451f\": container with ID starting with 40e6478512dc6ecc17c3f3ad32a70cefacd4d27ea594fbe86a0f57d970aa451f not found: ID does not exist" containerID="40e6478512dc6ecc17c3f3ad32a70cefacd4d27ea594fbe86a0f57d970aa451f"
Oct 30 04:15:19.250331 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.250261    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e6478512dc6ecc17c3f3ad32a70cefacd4d27ea594fbe86a0f57d970aa451f"} err="failed to get container status \"40e6478512dc6ecc17c3f3ad32a70cefacd4d27ea594fbe86a0f57d970aa451f\": rpc error: code = NotFound desc = could not find container \"40e6478512dc6ecc17c3f3ad32a70cefacd4d27ea594fbe86a0f57d970aa451f\": container with ID starting with 40e6478512dc6ecc17c3f3ad32a70cefacd4d27ea594fbe86a0f57d970aa451f not found: ID does not exist"
Oct 30 04:15:19.250331 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.250282    4960 scope.go:122] "RemoveContainer" containerID="ba7260c0ff3a2ad81c4dc02bff000d1d7604ffbbc3faf34cfeaa4b96eeff7489"
Oct 30 04:15:19.258081 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.258058    4960 scope.go:122] "RemoveContainer" containerID="507d5988471a128fb415479630790fe79ab1f9723c388bc3163c1fdf0b7e5521"
Oct 30 04:15:19.265415 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.265390    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:19.268156 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.268137    4960 scope.go:122] "RemoveContainer" containerID="ba7260c0ff3a2ad81c4dc02bff000d1d7604ffbbc3faf34cfeaa4b96eeff7489"
Oct 30 04:15:19.268478 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:15:19.268452    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba7260c0ff3a2ad81c4dc02bff000d1d7604ffbbc3faf34cfeaa4b96eeff7489\": container with ID starting with ba7260c0ff3a2ad81c4dc02bff000d1d7604ffbbc3faf34cfeaa4b96eeff7489 not found: ID does not exist" containerID="ba7260c0ff3a2ad81c4dc02bff000d1d7604ffbbc3faf34cfeaa4b96eeff7489"
Oct 30 04:15:19.268586 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.268487    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba7260c0ff3a2ad81c4dc02bff000d1d7604ffbbc3faf34cfeaa4b96eeff7489"} err="failed to get container status \"ba7260c0ff3a2ad81c4dc02bff000d1d7604ffbbc3faf34cfeaa4b96eeff7489\": rpc error: code = NotFound desc = could not find container \"ba7260c0ff3a2ad81c4dc02bff000d1d7604ffbbc3faf34cfeaa4b96eeff7489\": container with ID starting with ba7260c0ff3a2ad81c4dc02bff000d1d7604ffbbc3faf34cfeaa4b96eeff7489 not found: ID does not exist"
Oct 30 04:15:19.268586 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.268508    4960 scope.go:122] "RemoveContainer" containerID="507d5988471a128fb415479630790fe79ab1f9723c388bc3163c1fdf0b7e5521"
Oct 30 04:15:19.268799 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:15:19.268779    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507d5988471a128fb415479630790fe79ab1f9723c388bc3163c1fdf0b7e5521\": container with ID starting with 507d5988471a128fb415479630790fe79ab1f9723c388bc3163c1fdf0b7e5521 not found: ID does not exist" containerID="507d5988471a128fb415479630790fe79ab1f9723c388bc3163c1fdf0b7e5521"
Oct 30 04:15:19.268869 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.268811    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507d5988471a128fb415479630790fe79ab1f9723c388bc3163c1fdf0b7e5521"} err="failed to get container status \"507d5988471a128fb415479630790fe79ab1f9723c388bc3163c1fdf0b7e5521\": rpc error: code = NotFound desc = could not find container \"507d5988471a128fb415479630790fe79ab1f9723c388bc3163c1fdf0b7e5521\": container with ID starting with 507d5988471a128fb415479630790fe79ab1f9723c388bc3163c1fdf0b7e5521 not found: ID does not exist"
Oct 30 04:15:19.268869 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.268831    4960 scope.go:122] "RemoveContainer" containerID="ba7260c0ff3a2ad81c4dc02bff000d1d7604ffbbc3faf34cfeaa4b96eeff7489"
Oct 30 04:15:19.269083 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.269063    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba7260c0ff3a2ad81c4dc02bff000d1d7604ffbbc3faf34cfeaa4b96eeff7489"} err="failed to get container status \"ba7260c0ff3a2ad81c4dc02bff000d1d7604ffbbc3faf34cfeaa4b96eeff7489\": rpc error: code = NotFound desc = could not find container \"ba7260c0ff3a2ad81c4dc02bff000d1d7604ffbbc3faf34cfeaa4b96eeff7489\": container with ID starting with ba7260c0ff3a2ad81c4dc02bff000d1d7604ffbbc3faf34cfeaa4b96eeff7489 not found: ID does not exist"
Oct 30 04:15:19.269083 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.269083    4960 scope.go:122] "RemoveContainer" containerID="507d5988471a128fb415479630790fe79ab1f9723c388bc3163c1fdf0b7e5521"
Oct 30 04:15:19.269691 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.269664    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507d5988471a128fb415479630790fe79ab1f9723c388bc3163c1fdf0b7e5521"} err="failed to get container status \"507d5988471a128fb415479630790fe79ab1f9723c388bc3163c1fdf0b7e5521\": rpc error: code = NotFound desc = could not find container \"507d5988471a128fb415479630790fe79ab1f9723c388bc3163c1fdf0b7e5521\": container with ID starting with 507d5988471a128fb415479630790fe79ab1f9723c388bc3163c1fdf0b7e5521 not found: ID does not exist"
Oct 30 04:15:19.269785 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.269692    4960 scope.go:122] "RemoveContainer" containerID="d6eecf6272364353910c06209edf2df40f6e9c9a50de77b18b7cff6ddd4ea8e7"
Oct 30 04:15:19.278673 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.278658    4960 scope.go:122] "RemoveContainer" containerID="d6eecf6272364353910c06209edf2df40f6e9c9a50de77b18b7cff6ddd4ea8e7"
Oct 30 04:15:19.278979 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:15:19.278954    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6eecf6272364353910c06209edf2df40f6e9c9a50de77b18b7cff6ddd4ea8e7\": container with ID starting with d6eecf6272364353910c06209edf2df40f6e9c9a50de77b18b7cff6ddd4ea8e7 not found: ID does not exist" containerID="d6eecf6272364353910c06209edf2df40f6e9c9a50de77b18b7cff6ddd4ea8e7"
Oct 30 04:15:19.279053 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.278983    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6eecf6272364353910c06209edf2df40f6e9c9a50de77b18b7cff6ddd4ea8e7"} err="failed to get container status \"d6eecf6272364353910c06209edf2df40f6e9c9a50de77b18b7cff6ddd4ea8e7\": rpc error: code = NotFound desc = could not find container \"d6eecf6272364353910c06209edf2df40f6e9c9a50de77b18b7cff6ddd4ea8e7\": container with ID starting with d6eecf6272364353910c06209edf2df40f6e9c9a50de77b18b7cff6ddd4ea8e7 not found: ID does not exist"
Oct 30 04:15:19.283594 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.283571    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"]
Oct 30 04:15:19.288545 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.288498    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-sfftq"]
Oct 30 04:15:19.324377 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.324358    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b"]
Oct 30 04:15:19.328007 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.327990    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-s424b"]
Oct 30 04:15:19.331857 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.331838    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:19.363270 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.363252    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"]
Oct 30 04:15:19.366336 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.366317    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") "
Oct 30 04:15:19.366411 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.366369    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-secret-prometheus-web-tls-ca\") pod \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") "
Oct 30 04:15:19.366411 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.366406    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-config-out\" (UniqueName: \"kubernetes.io/empty-dir/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-config-out\") pod \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") "
Oct 30 04:15:19.366521 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.366507    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") "
Oct 30 04:15:19.366566 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.366552    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-config\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-config\") pod \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") "
Oct 30 04:15:19.366609 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.366585    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-tls-assets\" (UniqueName: \"kubernetes.io/projected/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-tls-assets\") pod \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") "
Oct 30 04:15:19.366657 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.366617    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") "
Oct 30 04:15:19.366729 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.366654    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") "
Oct 30 04:15:19.366729 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.366687    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") "
Oct 30 04:15:19.366843 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.366768    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-thanos-prometheus-http-client-file\") pod \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") "
Oct 30 04:15:19.366843 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.366797    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-kube-api-access-bsg7j\" (UniqueName: \"kubernetes.io/projected/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-kube-api-access-bsg7j\") pod \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") "
Oct 30 04:15:19.366843 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.366825    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") "
Oct 30 04:15:19.367007 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.366863    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config\") pod \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\" (UID: \"28c36ec9-29c3-4b59-a82e-7da81e0e6a08\") "
Oct 30 04:15:19.367758 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.367734    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-2" pod "28c36ec9-29c3-4b59-a82e-7da81e0e6a08" (UID: "28c36ec9-29c3-4b59-a82e-7da81e0e6a08"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:19.368563 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.368514    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-0" pod "28c36ec9-29c3-4b59-a82e-7da81e0e6a08" (UID: "28c36ec9-29c3-4b59-a82e-7da81e0e6a08"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:19.368971 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.368943    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-1" pod "28c36ec9-29c3-4b59-a82e-7da81e0e6a08" (UID: "28c36ec9-29c3-4b59-a82e-7da81e0e6a08"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:19.370401 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.370373    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-secret-prometheus-web-tls-ca" pod "28c36ec9-29c3-4b59-a82e-7da81e0e6a08" (UID: "28c36ec9-29c3-4b59-a82e-7da81e0e6a08"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:19.370691 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.370669    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-config-out" pod "28c36ec9-29c3-4b59-a82e-7da81e0e6a08" (UID: "28c36ec9-29c3-4b59-a82e-7da81e0e6a08"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:15:19.371119 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.371099    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-xnsgw"]
Oct 30 04:15:19.371347 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.371137    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-config" pod "28c36ec9-29c3-4b59-a82e-7da81e0e6a08" (UID: "28c36ec9-29c3-4b59-a82e-7da81e0e6a08"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:19.371347 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.371249    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-tls-assets" pod "28c36ec9-29c3-4b59-a82e-7da81e0e6a08" (UID: "28c36ec9-29c3-4b59-a82e-7da81e0e6a08"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:19.371347 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.371295    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-kube-api-access-bsg7j" pod "28c36ec9-29c3-4b59-a82e-7da81e0e6a08" (UID: "28c36ec9-29c3-4b59-a82e-7da81e0e6a08"). InnerVolumeSpecName "kube-api-access-bsg7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:19.371613 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.371593    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "28c36ec9-29c3-4b59-a82e-7da81e0e6a08" (UID: "28c36ec9-29c3-4b59-a82e-7da81e0e6a08"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:19.371679 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.371617    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "28c36ec9-29c3-4b59-a82e-7da81e0e6a08" (UID: "28c36ec9-29c3-4b59-a82e-7da81e0e6a08"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:19.372972 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.372953    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-thanos-prometheus-http-client-file" pod "28c36ec9-29c3-4b59-a82e-7da81e0e6a08" (UID: "28c36ec9-29c3-4b59-a82e-7da81e0e6a08"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:19.373467 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.373447    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2" pod "28c36ec9-29c3-4b59-a82e-7da81e0e6a08" (UID: "28c36ec9-29c3-4b59-a82e-7da81e0e6a08"). InnerVolumeSpecName "pvc-72ca8347-922a-464b-8594-dfac31ff3c9d". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:15:19.376863 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.376842    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:19.383044 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.383024    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config" pod "28c36ec9-29c3-4b59-a82e-7da81e0e6a08" (UID: "28c36ec9-29c3-4b59-a82e-7da81e0e6a08"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:19.467507 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.467479    4960 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-secret-prometheus-web-tls-ca\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.467507 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.467504    4960 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-config-out\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.467742 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.467531    4960 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") on node \"ip-10-0-128-241.ec2.internal\" "
Oct 30 04:15:19.467742 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.467541    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.467742 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.467550    4960 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-tls-assets\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.467742 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.467560    4960 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.467742 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.467569    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.467742 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.467579    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.467742 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.467589    4960 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.467742 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.467597    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsg7j\" (UniqueName: \"kubernetes.io/projected/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-kube-api-access-bsg7j\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.467742 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.467607    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.467742 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.467616    4960 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.467742 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.467624    4960 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/28c36ec9-29c3-4b59-a82e-7da81e0e6a08-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.481392 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.481369    4960 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-72ca8347-922a-464b-8594-dfac31ff3c9d" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2") on node "ip-10-0-128-241.ec2.internal"
Oct 30 04:15:19.503165 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.503136    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02b485e3-c53a-4126-8854-1639da65cb05" path="/var/lib/kubelet/pods/02b485e3-c53a-4126-8854-1639da65cb05/volumes"
Oct 30 04:15:19.503700 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.503682    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07b685f5-e45c-4c9c-a170-6d26a643341b" path="/var/lib/kubelet/pods/07b685f5-e45c-4c9c-a170-6d26a643341b/volumes"
Oct 30 04:15:19.504208 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.504172    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73fbee4b-fa9b-47e7-b350-427e832dd16c" path="/var/lib/kubelet/pods/73fbee4b-fa9b-47e7-b350-427e832dd16c/volumes"
Oct 30 04:15:19.505137 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.505118    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae264e2d-42ab-435c-a4f4-df406a2c9ee8" path="/var/lib/kubelet/pods/ae264e2d-42ab-435c-a4f4-df406a2c9ee8/volumes"
Oct 30 04:15:19.568831 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.568806    4960 reconciler_common.go:299] "Volume detached for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"csi-2479454746a6578572f425530fc81bfe97655fb31272a4af03192f10a097ebe5\""
Oct 30 04:15:19.715461 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.715420    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:19.834791 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.834764    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:19.937624 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:19.937595    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:20.141058 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.140987    4960 generic.go:356] "Generic (PLEG): container finished" podID="2c4c28df-ac78-4d9e-99cb-26b725bc9c79" containerID="8f3c52c589bead205a0a30b44746f898cba39f2713babafacb96266a953cdb9c" exitCode=0
Oct 30 04:15:20.141213 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.141055    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2c4c28df-ac78-4d9e-99cb-26b725bc9c79","Type":"ContainerDied","Data":"8f3c52c589bead205a0a30b44746f898cba39f2713babafacb96266a953cdb9c"}
Oct 30 04:15:20.147279 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.147252    4960 generic.go:356] "Generic (PLEG): container finished" podID="28c36ec9-29c3-4b59-a82e-7da81e0e6a08" containerID="682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0" exitCode=0
Oct 30 04:15:20.147279 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.147278    4960 generic.go:356] "Generic (PLEG): container finished" podID="28c36ec9-29c3-4b59-a82e-7da81e0e6a08" containerID="bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd" exitCode=0
Oct 30 04:15:20.147458 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.147288    4960 generic.go:356] "Generic (PLEG): container finished" podID="28c36ec9-29c3-4b59-a82e-7da81e0e6a08" containerID="a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451" exitCode=0
Oct 30 04:15:20.147458 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.147324    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"28c36ec9-29c3-4b59-a82e-7da81e0e6a08","Type":"ContainerDied","Data":"682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0"}
Oct 30 04:15:20.147458 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.147342    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:20.147458 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.147374    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"28c36ec9-29c3-4b59-a82e-7da81e0e6a08","Type":"ContainerDied","Data":"bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd"}
Oct 30 04:15:20.147458 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.147398    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"28c36ec9-29c3-4b59-a82e-7da81e0e6a08","Type":"ContainerDied","Data":"a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451"}
Oct 30 04:15:20.147458 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.147419    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"28c36ec9-29c3-4b59-a82e-7da81e0e6a08","Type":"ContainerDied","Data":"0743f414becbcdbd0a17044789e5581c893fad9c33205969ed2dddea9b6f6a45"}
Oct 30 04:15:20.147458 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.147451    4960 scope.go:122] "RemoveContainer" containerID="682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0"
Oct 30 04:15:20.156685 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.156666    4960 scope.go:122] "RemoveContainer" containerID="bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd"
Oct 30 04:15:20.165052 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.165036    4960 scope.go:122] "RemoveContainer" containerID="a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451"
Oct 30 04:15:20.172543 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.172514    4960 scope.go:122] "RemoveContainer" containerID="1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689"
Oct 30 04:15:20.179796 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.179766    4960 scope.go:122] "RemoveContainer" containerID="682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0"
Oct 30 04:15:20.180075 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:15:20.180058    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0\": container with ID starting with 682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0 not found: ID does not exist" containerID="682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0"
Oct 30 04:15:20.180154 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.180087    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0"} err="failed to get container status \"682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0\": rpc error: code = NotFound desc = could not find container \"682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0\": container with ID starting with 682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0 not found: ID does not exist"
Oct 30 04:15:20.180154 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.180111    4960 scope.go:122] "RemoveContainer" containerID="bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd"
Oct 30 04:15:20.180405 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:15:20.180388    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd\": container with ID starting with bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd not found: ID does not exist" containerID="bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd"
Oct 30 04:15:20.180443 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.180412    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd"} err="failed to get container status \"bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd\": rpc error: code = NotFound desc = could not find container \"bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd\": container with ID starting with bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd not found: ID does not exist"
Oct 30 04:15:20.180443 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.180428    4960 scope.go:122] "RemoveContainer" containerID="a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451"
Oct 30 04:15:20.180694 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:15:20.180674    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451\": container with ID starting with a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451 not found: ID does not exist" containerID="a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451"
Oct 30 04:15:20.180738 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.180702    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451"} err="failed to get container status \"a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451\": rpc error: code = NotFound desc = could not find container \"a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451\": container with ID starting with a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451 not found: ID does not exist"
Oct 30 04:15:20.180738 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.180721    4960 scope.go:122] "RemoveContainer" containerID="1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689"
Oct 30 04:15:20.180989 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:15:20.180969    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689\": container with ID starting with 1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689 not found: ID does not exist" containerID="1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689"
Oct 30 04:15:20.181037 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.180995    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689"} err="failed to get container status \"1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689\": rpc error: code = NotFound desc = could not find container \"1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689\": container with ID starting with 1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689 not found: ID does not exist"
Oct 30 04:15:20.181037 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.181009    4960 scope.go:122] "RemoveContainer" containerID="682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0"
Oct 30 04:15:20.181258 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.181230    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0"} err="failed to get container status \"682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0\": rpc error: code = NotFound desc = could not find container \"682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0\": container with ID starting with 682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0 not found: ID does not exist"
Oct 30 04:15:20.181361 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.181257    4960 scope.go:122] "RemoveContainer" containerID="bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd"
Oct 30 04:15:20.181509 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.181494    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd"} err="failed to get container status \"bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd\": rpc error: code = NotFound desc = could not find container \"bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd\": container with ID starting with bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd not found: ID does not exist"
Oct 30 04:15:20.181559 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.181510    4960 scope.go:122] "RemoveContainer" containerID="a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451"
Oct 30 04:15:20.181758 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.181733    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451"} err="failed to get container status \"a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451\": rpc error: code = NotFound desc = could not find container \"a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451\": container with ID starting with a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451 not found: ID does not exist"
Oct 30 04:15:20.181834 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.181760    4960 scope.go:122] "RemoveContainer" containerID="1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689"
Oct 30 04:15:20.182039 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.182015    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689"} err="failed to get container status \"1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689\": rpc error: code = NotFound desc = could not find container \"1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689\": container with ID starting with 1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689 not found: ID does not exist"
Oct 30 04:15:20.182116 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.182039    4960 scope.go:122] "RemoveContainer" containerID="682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0"
Oct 30 04:15:20.182436 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.182410    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0"} err="failed to get container status \"682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0\": rpc error: code = NotFound desc = could not find container \"682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0\": container with ID starting with 682613b9d6d658a12d192da76f1f55571fcb42f235d456a292f846027d515ad0 not found: ID does not exist"
Oct 30 04:15:20.182436 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.182435    4960 scope.go:122] "RemoveContainer" containerID="bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd"
Oct 30 04:15:20.182716 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.182686    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd"} err="failed to get container status \"bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd\": rpc error: code = NotFound desc = could not find container \"bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd\": container with ID starting with bf17ebb0a9026180c0ff03f89ae74b19f1b0e6f9158936267c52f9227b1a9ebd not found: ID does not exist"
Oct 30 04:15:20.182716 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.182707    4960 scope.go:122] "RemoveContainer" containerID="a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451"
Oct 30 04:15:20.182969 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.182953    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451"} err="failed to get container status \"a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451\": rpc error: code = NotFound desc = could not find container \"a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451\": container with ID starting with a27e3b03eae3337e6fbb359b61519eb1bb93f5aa96ccdc4ddd4bc3262e0ff451 not found: ID does not exist"
Oct 30 04:15:20.183006 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.182969    4960 scope.go:122] "RemoveContainer" containerID="1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689"
Oct 30 04:15:20.183215 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.183194    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689"} err="failed to get container status \"1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689\": rpc error: code = NotFound desc = could not find container \"1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689\": container with ID starting with 1f64cdf245e78ba0234aed4519c9cc0d099c801be771356bda09829f64eb7689 not found: ID does not exist"
Oct 30 04:15:20.202978 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.202957    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:15:20.216494 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.216470    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:15:20.659893 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.659865    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:20.778788 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.778753    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:20.884151 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:20.884121    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:21.507693 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:21.507651    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28c36ec9-29c3-4b59-a82e-7da81e0e6a08" path="/var/lib/kubelet/pods/28c36ec9-29c3-4b59-a82e-7da81e0e6a08/volumes"
Oct 30 04:15:22.998832 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:22.998796    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="2c4c28df-ac78-4d9e-99cb-26b725bc9c79" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.124:9093/-/ready\": dial tcp 10.132.0.124:9093: connect: connection refused"
Oct 30 04:15:23.953863 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:15:23.953830    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:15:25.325040 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.325007    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:25.435727 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.435697    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:25.545307 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.545278    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:25.679937 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.679843    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:25.740862 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.740826    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"]
Oct 30 04:15:25.741237 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741220    4960 cpu_manager.go:414] "containerMap: removing container" podUID="02b485e3-c53a-4126-8854-1639da65cb05" containerName="kube-rbac-proxy"
Oct 30 04:15:25.741344 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741240    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b485e3-c53a-4126-8854-1639da65cb05" containerName="kube-rbac-proxy"
Oct 30 04:15:25.741344 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741275    4960 cpu_manager.go:414] "containerMap: removing container" podUID="73fbee4b-fa9b-47e7-b350-427e832dd16c" containerName="kube-rbac-proxy"
Oct 30 04:15:25.741344 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741283    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73fbee4b-fa9b-47e7-b350-427e832dd16c" containerName="kube-rbac-proxy"
Oct 30 04:15:25.741344 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741299    4960 cpu_manager.go:414] "containerMap: removing container" podUID="ae264e2d-42ab-435c-a4f4-df406a2c9ee8" containerName="kube-rbac-proxy"
Oct 30 04:15:25.741344 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741306    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae264e2d-42ab-435c-a4f4-df406a2c9ee8" containerName="kube-rbac-proxy"
Oct 30 04:15:25.741344 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741317    4960 cpu_manager.go:414] "containerMap: removing container" podUID="07b685f5-e45c-4c9c-a170-6d26a643341b" containerName="korrel8r"
Oct 30 04:15:25.741344 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741325    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b685f5-e45c-4c9c-a170-6d26a643341b" containerName="korrel8r"
Oct 30 04:15:25.741344 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741340    4960 cpu_manager.go:414] "containerMap: removing container" podUID="28c36ec9-29c3-4b59-a82e-7da81e0e6a08" containerName="init-config-reloader"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741347    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c36ec9-29c3-4b59-a82e-7da81e0e6a08" containerName="init-config-reloader"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741360    4960 cpu_manager.go:414] "containerMap: removing container" podUID="28c36ec9-29c3-4b59-a82e-7da81e0e6a08" containerName="thanos-sidecar"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741367    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c36ec9-29c3-4b59-a82e-7da81e0e6a08" containerName="thanos-sidecar"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741380    4960 cpu_manager.go:414] "containerMap: removing container" podUID="ae264e2d-42ab-435c-a4f4-df406a2c9ee8" containerName="prom-label-proxy"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741388    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae264e2d-42ab-435c-a4f4-df406a2c9ee8" containerName="prom-label-proxy"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741398    4960 cpu_manager.go:414] "containerMap: removing container" podUID="28c36ec9-29c3-4b59-a82e-7da81e0e6a08" containerName="prometheus"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741405    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c36ec9-29c3-4b59-a82e-7da81e0e6a08" containerName="prometheus"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741415    4960 cpu_manager.go:414] "containerMap: removing container" podUID="73fbee4b-fa9b-47e7-b350-427e832dd16c" containerName="prom-label-proxy"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741423    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73fbee4b-fa9b-47e7-b350-427e832dd16c" containerName="prom-label-proxy"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741435    4960 cpu_manager.go:414] "containerMap: removing container" podUID="28c36ec9-29c3-4b59-a82e-7da81e0e6a08" containerName="config-reloader"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741443    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c36ec9-29c3-4b59-a82e-7da81e0e6a08" containerName="config-reloader"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741457    4960 cpu_manager.go:414] "containerMap: removing container" podUID="66922329-3872-4cd5-9783-a032a7d34d2b" containerName="otc-container"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741463    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66922329-3872-4cd5-9783-a032a7d34d2b" containerName="otc-container"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741573    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b485e3-c53a-4126-8854-1639da65cb05" containerName="kube-rbac-proxy"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741590    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="73fbee4b-fa9b-47e7-b350-427e832dd16c" containerName="kube-rbac-proxy"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741599    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae264e2d-42ab-435c-a4f4-df406a2c9ee8" containerName="kube-rbac-proxy"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741606    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b685f5-e45c-4c9c-a170-6d26a643341b" containerName="korrel8r"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741616    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c36ec9-29c3-4b59-a82e-7da81e0e6a08" containerName="thanos-sidecar"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741625    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae264e2d-42ab-435c-a4f4-df406a2c9ee8" containerName="prom-label-proxy"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741634    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c36ec9-29c3-4b59-a82e-7da81e0e6a08" containerName="prometheus"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741642    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="73fbee4b-fa9b-47e7-b350-427e832dd16c" containerName="prom-label-proxy"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741651    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c36ec9-29c3-4b59-a82e-7da81e0e6a08" containerName="config-reloader"
Oct 30 04:15:25.741759 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.741659    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="66922329-3872-4cd5-9783-a032a7d34d2b" containerName="otc-container"
Oct 30 04:15:25.746849 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.746827    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"
Oct 30 04:15:25.764665 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.764644    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"]
Oct 30 04:15:25.790217 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.790151    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:25.820288 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.820262    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/87d89dab-459a-499a-b417-7079799bcf56-tls\") pod \"korrel8r-5cbcc89b8f-nm8rj\" (UID: \"87d89dab-459a-499a-b417-7079799bcf56\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"
Oct 30 04:15:25.820379 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.820309    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/87d89dab-459a-499a-b417-7079799bcf56-config\") pod \"korrel8r-5cbcc89b8f-nm8rj\" (UID: \"87d89dab-459a-499a-b417-7079799bcf56\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"
Oct 30 04:15:25.820415 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.820383    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7rpf\" (UniqueName: \"kubernetes.io/projected/87d89dab-459a-499a-b417-7079799bcf56-kube-api-access-n7rpf\") pod \"korrel8r-5cbcc89b8f-nm8rj\" (UID: \"87d89dab-459a-499a-b417-7079799bcf56\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"
Oct 30 04:15:25.838550 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.838529    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\""
Oct 30 04:15:25.838648 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.838580    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\""
Oct 30 04:15:25.838762 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.838747    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-b26ll\""
Oct 30 04:15:25.920746 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.920717    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7rpf\" (UniqueName: \"kubernetes.io/projected/87d89dab-459a-499a-b417-7079799bcf56-kube-api-access-n7rpf\") pod \"korrel8r-5cbcc89b8f-nm8rj\" (UID: \"87d89dab-459a-499a-b417-7079799bcf56\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"
Oct 30 04:15:25.920839 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.920776    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/87d89dab-459a-499a-b417-7079799bcf56-tls\") pod \"korrel8r-5cbcc89b8f-nm8rj\" (UID: \"87d89dab-459a-499a-b417-7079799bcf56\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"
Oct 30 04:15:25.920912 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.920838    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/87d89dab-459a-499a-b417-7079799bcf56-config\") pod \"korrel8r-5cbcc89b8f-nm8rj\" (UID: \"87d89dab-459a-499a-b417-7079799bcf56\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"
Oct 30 04:15:25.921622 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.921601    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/87d89dab-459a-499a-b417-7079799bcf56-config\") pod \"korrel8r-5cbcc89b8f-nm8rj\" (UID: \"87d89dab-459a-499a-b417-7079799bcf56\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"
Oct 30 04:15:25.923313 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.923289    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/87d89dab-459a-499a-b417-7079799bcf56-tls\") pod \"korrel8r-5cbcc89b8f-nm8rj\" (UID: \"87d89dab-459a-499a-b417-7079799bcf56\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"
Oct 30 04:15:25.941348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:25.941300    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7rpf\" (UniqueName: \"kubernetes.io/projected/87d89dab-459a-499a-b417-7079799bcf56-kube-api-access-n7rpf\") pod \"korrel8r-5cbcc89b8f-nm8rj\" (UID: \"87d89dab-459a-499a-b417-7079799bcf56\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"
Oct 30 04:15:26.059165 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:26.059137    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"
Oct 30 04:15:26.248475 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:26.248401    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:26.348319 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:26.348293    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"]
Oct 30 04:15:26.349399 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:15:26.349370    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d89dab_459a_499a_b417_7079799bcf56.slice/crio-ae20aed180515f0dc3a65b2b56d4f285ea2b3cde01a1efdcd1f9b597e1c4e80e WatchSource:0}: Error finding container ae20aed180515f0dc3a65b2b56d4f285ea2b3cde01a1efdcd1f9b597e1c4e80e: Status 404 returned error can't find the container with id ae20aed180515f0dc3a65b2b56d4f285ea2b3cde01a1efdcd1f9b597e1c4e80e
Oct 30 04:15:26.379948 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:26.379928    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:26.579314 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:26.579280    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:27.135533 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.135503    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:15:27.139083 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.139058    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.159984 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.159956    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:15:27.175097 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.175068    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:27.182548 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.182521    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:15:27.186103 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.186073    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj" event={"ID":"87d89dab-459a-499a-b417-7079799bcf56","Type":"ContainerStarted","Data":"83073ec0201f8179f21f0fc796a209df04a94aa6a7ce9b127a5689138dadc186"}
Oct 30 04:15:27.186254 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.186127    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj" event={"ID":"87d89dab-459a-499a-b417-7079799bcf56","Type":"ContainerStarted","Data":"ae20aed180515f0dc3a65b2b56d4f285ea2b3cde01a1efdcd1f9b597e1c4e80e"}
Oct 30 04:15:27.186254 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.186238    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"
Oct 30 04:15:27.220003 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.219979    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:15:27.220164 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.220048    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:15:27.220260 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.220244    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-m9vgz\""
Oct 30 04:15:27.220396 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.220372    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:15:27.220448 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.220437    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:15:27.231508 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.231485    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.231589 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.231518    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.231640 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.231589    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.231694 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.231681    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.231735 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.231707    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7dzw\" (UniqueName: \"kubernetes.io/projected/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-kube-api-access-n7dzw\") pod \"data-science-collector-collector-1\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.231735 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.231730    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.255866 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.255817    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj" podStartSLOduration=2.255801173 podStartE2EDuration="2.255801173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:27.255049554 +0000 UTC m=+5607.904097977" watchObservedRunningTime="2026-10-30 04:15:27.255801173 +0000 UTC m=+5607.904849595"
Oct 30 04:15:27.332688 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.332652    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.332979 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.332701    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.332979 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.332836    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.332979 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.332872    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.332979 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.332929    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.332979 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.332952    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7dzw\" (UniqueName: \"kubernetes.io/projected/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-kube-api-access-n7dzw\") pod \"data-science-collector-collector-1\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.333665 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.333639    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.335387 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.335365    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.335881 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.335861    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.336093 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.336070    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.336136 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.336082    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.363372 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.363352    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7dzw\" (UniqueName: \"kubernetes.io/projected/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-kube-api-access-n7dzw\") pod \"data-science-collector-collector-1\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.388253 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.388199    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:27.454643 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.454617    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:27.510600 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.510554    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:27.749809 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.749786    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:15:27.751242 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:15:27.751209    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd958dfa3_1c20_4ac1_b00a_b67cecf0ee63.slice/crio-44778e39910d519dbe9dade9682ea2f32fcf1501382d57d6baa527a39a01e84e WatchSource:0}: Error finding container 44778e39910d519dbe9dade9682ea2f32fcf1501382d57d6baa527a39a01e84e: Status 404 returned error can't find the container with id 44778e39910d519dbe9dade9682ea2f32fcf1501382d57d6baa527a39a01e84e
Oct 30 04:15:27.753425 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.753390    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:15:27.753533 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.753455    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:15:27.999047 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:27.998975    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="2c4c28df-ac78-4d9e-99cb-26b725bc9c79" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.124:9093/-/ready\": dial tcp 10.132.0.124:9093: connect: connection refused"
Oct 30 04:15:28.192129 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:28.192086    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63","Type":"ContainerStarted","Data":"d7a038355120261abce1634bd6c5cffc4ee54bdcd2134793e1c2d131e45e53ba"}
Oct 30 04:15:28.192129 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:28.192137    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63","Type":"ContainerStarted","Data":"44778e39910d519dbe9dade9682ea2f32fcf1501382d57d6baa527a39a01e84e"}
Oct 30 04:15:28.234571 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:28.234524    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.234511355 podStartE2EDuration="1.234511355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:28.233290226 +0000 UTC m=+5608.882338648" watchObservedRunningTime="2026-10-30 04:15:28.234511355 +0000 UTC m=+5608.883559776"
Oct 30 04:15:29.200469 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.200436    4960 generic.go:356] "Generic (PLEG): container finished" podID="2c4c28df-ac78-4d9e-99cb-26b725bc9c79" containerID="a7e939893a308f5f0c7f015d57be414b45e8a20ccad4f10857138cbc8a51d5c4" exitCode=0
Oct 30 04:15:29.200936 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.200509    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2c4c28df-ac78-4d9e-99cb-26b725bc9c79","Type":"ContainerDied","Data":"a7e939893a308f5f0c7f015d57be414b45e8a20ccad4f10857138cbc8a51d5c4"}
Oct 30 04:15:29.291910 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.291882    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:29.350323 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.350295    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-config-volume\" (UniqueName: \"kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-config-volume\") pod \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") "
Oct 30 04:15:29.350593 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.350342    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-web-config\" (UniqueName: \"kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-web-config\") pod \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") "
Oct 30 04:15:29.350593 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.350382    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-tls-assets\" (UniqueName: \"kubernetes.io/projected/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-tls-assets\") pod \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") "
Oct 30 04:15:29.350593 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.350414    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-kube-api-access-c68gp\" (UniqueName: \"kubernetes.io/projected/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-kube-api-access-c68gp\") pod \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") "
Oct 30 04:15:29.350593 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.350578    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-alertmanager-data-science-monitoringstack-db\") pod \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") "
Oct 30 04:15:29.350881 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.350613    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-cluster-tls-config\") pod \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") "
Oct 30 04:15:29.350881 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.350828    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-config-out\") pod \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\" (UID: \"2c4c28df-ac78-4d9e-99cb-26b725bc9c79\") "
Oct 30 04:15:29.353052 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.351701    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-alertmanager-data-science-monitoringstack-db" pod "2c4c28df-ac78-4d9e-99cb-26b725bc9c79" (UID: "2c4c28df-ac78-4d9e-99cb-26b725bc9c79"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:15:29.353934 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.353905    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-config-volume" pod "2c4c28df-ac78-4d9e-99cb-26b725bc9c79" (UID: "2c4c28df-ac78-4d9e-99cb-26b725bc9c79"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:29.354947 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.354925    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-kube-api-access-c68gp" pod "2c4c28df-ac78-4d9e-99cb-26b725bc9c79" (UID: "2c4c28df-ac78-4d9e-99cb-26b725bc9c79"). InnerVolumeSpecName "kube-api-access-c68gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:29.355377 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.355356    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-tls-assets" pod "2c4c28df-ac78-4d9e-99cb-26b725bc9c79" (UID: "2c4c28df-ac78-4d9e-99cb-26b725bc9c79"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:29.356520 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.356494    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-config-out" pod "2c4c28df-ac78-4d9e-99cb-26b725bc9c79" (UID: "2c4c28df-ac78-4d9e-99cb-26b725bc9c79"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:15:29.359678 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.359651    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-cluster-tls-config" pod "2c4c28df-ac78-4d9e-99cb-26b725bc9c79" (UID: "2c4c28df-ac78-4d9e-99cb-26b725bc9c79"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:29.367389 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.367302    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-web-config" pod "2c4c28df-ac78-4d9e-99cb-26b725bc9c79" (UID: "2c4c28df-ac78-4d9e-99cb-26b725bc9c79"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:29.451970 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.451946    4960 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-config-volume\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:29.451970 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.451971    4960 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-web-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:29.452146 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.451980    4960 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-tls-assets\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:29.452146 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.451988    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c68gp\" (UniqueName: \"kubernetes.io/projected/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-kube-api-access-c68gp\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:29.452146 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.451997    4960 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:29.452146 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.452008    4960 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-cluster-tls-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:29.452146 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:29.452016    4960 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c4c28df-ac78-4d9e-99cb-26b725bc9c79-config-out\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:30.151189 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.151155    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:30.210573 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.210543    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2c4c28df-ac78-4d9e-99cb-26b725bc9c79","Type":"ContainerDied","Data":"a3ffd3cbed07d0f723705fb497cade1d0e6a383e4e93670b43e377fc3a28764d"}
Oct 30 04:15:30.210972 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.210613    4960 scope.go:122] "RemoveContainer" containerID="8f3c52c589bead205a0a30b44746f898cba39f2713babafacb96266a953cdb9c"
Oct 30 04:15:30.210972 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.210558    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.219256 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.219238    4960 scope.go:122] "RemoveContainer" containerID="a7e939893a308f5f0c7f015d57be414b45e8a20ccad4f10857138cbc8a51d5c4"
Oct 30 04:15:30.227063 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.227047    4960 scope.go:122] "RemoveContainer" containerID="67ea680d183ed7f574586926e11e53c4a9b9e0c0d5dafde85eaa1ba3899b5ab1"
Oct 30 04:15:30.281881 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.281857    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:15:30.295226 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.295201    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:15:30.386820 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.386795    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:30.491121 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.491088    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:30.739922 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.739836    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:15:30.740201 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.740169    4960 cpu_manager.go:414] "containerMap: removing container" podUID="2c4c28df-ac78-4d9e-99cb-26b725bc9c79" containerName="config-reloader"
Oct 30 04:15:30.740294 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.740202    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c4c28df-ac78-4d9e-99cb-26b725bc9c79" containerName="config-reloader"
Oct 30 04:15:30.740294 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.740224    4960 cpu_manager.go:414] "containerMap: removing container" podUID="2c4c28df-ac78-4d9e-99cb-26b725bc9c79" containerName="init-config-reloader"
Oct 30 04:15:30.740294 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.740229    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c4c28df-ac78-4d9e-99cb-26b725bc9c79" containerName="init-config-reloader"
Oct 30 04:15:30.740294 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.740246    4960 cpu_manager.go:414] "containerMap: removing container" podUID="2c4c28df-ac78-4d9e-99cb-26b725bc9c79" containerName="alertmanager"
Oct 30 04:15:30.740294 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.740252    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c4c28df-ac78-4d9e-99cb-26b725bc9c79" containerName="alertmanager"
Oct 30 04:15:30.740552 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.740311    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c4c28df-ac78-4d9e-99cb-26b725bc9c79" containerName="alertmanager"
Oct 30 04:15:30.740552 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.740322    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c4c28df-ac78-4d9e-99cb-26b725bc9c79" containerName="config-reloader"
Oct 30 04:15:30.745496 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.745476    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.763059 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.763033    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\""
Oct 30 04:15:30.763222 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.763072    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\""
Oct 30 04:15:30.763880 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.763113    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:15:30.763880 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.763144    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\""
Oct 30 04:15:30.764077 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.763947    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-dwdtj\""
Oct 30 04:15:30.765342 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.765319    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:15:30.766457 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.766434    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:30.785936 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.785915    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:15:30.789525 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.789507    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.806487 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.806464    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 04:15:30.806602 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.806488    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 04:15:30.806682 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.806663    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 04:15:30.822801 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.822765    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:15:30.863301 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863241    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.863301 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863301    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.863597 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863331    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-8vqvg\""
Oct 30 04:15:30.863597 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863333    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.863597 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863350    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:15:30.863597 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863405    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.863597 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863410    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:15:30.863597 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863464    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 04:15:30.863597 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863469    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 04:15:30.863597 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863483    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c289ddb6-01c2-458e-ad4f-59dd07861347-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.863597 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863521    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 04:15:30.863597 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863516    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.863597 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863584    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.864002 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863613    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 04:15:30.864002 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863620    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6abe4c2b-e4a2-4fab-ab20-81eb92486337-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.864002 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863718    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c289ddb6-01c2-458e-ad4f-59dd07861347-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.864002 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863741    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.864002 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863761    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r6jx\" (UniqueName: \"kubernetes.io/projected/6abe4c2b-e4a2-4fab-ab20-81eb92486337-kube-api-access-7r6jx\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.864002 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863790    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.864002 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863878    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.864002 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863898    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n784q\" (UniqueName: \"kubernetes.io/projected/c289ddb6-01c2-458e-ad4f-59dd07861347-kube-api-access-n784q\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.864002 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863920    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6abe4c2b-e4a2-4fab-ab20-81eb92486337-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.864002 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863965    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.864002 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.863991    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.864381 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.864018    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6abe4c2b-e4a2-4fab-ab20-81eb92486337-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.864381 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.864054    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.885791 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.885761    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:30.964743 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.964710    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.964743 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.964742    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6abe4c2b-e4a2-4fab-ab20-81eb92486337-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.964928 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.964760    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c289ddb6-01c2-458e-ad4f-59dd07861347-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.964928 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.964776    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.964928 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.964800    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r6jx\" (UniqueName: \"kubernetes.io/projected/6abe4c2b-e4a2-4fab-ab20-81eb92486337-kube-api-access-7r6jx\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.964928 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.964827    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.964928 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.964860    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.964928 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.964884    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n784q\" (UniqueName: \"kubernetes.io/projected/c289ddb6-01c2-458e-ad4f-59dd07861347-kube-api-access-n784q\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.964928 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.964908    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6abe4c2b-e4a2-4fab-ab20-81eb92486337-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.965347 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.964933    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.965569 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.965549    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.965723 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.965702    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6abe4c2b-e4a2-4fab-ab20-81eb92486337-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.965943 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.965925    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.966086 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.966066    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.966258 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.966238    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6abe4c2b-e4a2-4fab-ab20-81eb92486337-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.966350 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.966244    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.966350 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.966301    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.966350 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.966330    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.966515 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.966384    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c289ddb6-01c2-458e-ad4f-59dd07861347-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.966515 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.966411    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.967308 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.967275    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.967404 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.967389    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.968308 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.968285    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.968580 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.968533    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.969263 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.969242    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c289ddb6-01c2-458e-ad4f-59dd07861347-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.969691 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.969662    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.970238 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.970025    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6abe4c2b-e4a2-4fab-ab20-81eb92486337-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.970238 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.970199    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.970866 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.970390    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.970866 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.970653    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.970866 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.970823    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.971080 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.970998    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6abe4c2b-e4a2-4fab-ab20-81eb92486337-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.971462 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.971438    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c289ddb6-01c2-458e-ad4f-59dd07861347-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.971539 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.971487    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:30.971539 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.971512    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.971655 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.971638    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.985542 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.985508    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r6jx\" (UniqueName: \"kubernetes.io/projected/6abe4c2b-e4a2-4fab-ab20-81eb92486337-kube-api-access-7r6jx\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:30.985984 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:30.985964    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n784q\" (UniqueName: \"kubernetes.io/projected/c289ddb6-01c2-458e-ad4f-59dd07861347-kube-api-access-n784q\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:31.062631 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:31.062577    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:31.377457 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:31.377427    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:15:31.389036 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:15:31.389006    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abe4c2b_e4a2_4fab_ab20_81eb92486337.slice/crio-0a4e116dcabdced0282940cf4aa9d650758e949d144265f1c605b1797b28376b WatchSource:0}: Error finding container 0a4e116dcabdced0282940cf4aa9d650758e949d144265f1c605b1797b28376b: Status 404 returned error can't find the container with id 0a4e116dcabdced0282940cf4aa9d650758e949d144265f1c605b1797b28376b
Oct 30 04:15:31.502221 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:31.502173    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c4c28df-ac78-4d9e-99cb-26b725bc9c79" path="/var/lib/kubelet/pods/2c4c28df-ac78-4d9e-99cb-26b725bc9c79/volumes"
Oct 30 04:15:32.223191 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:32.223149    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6abe4c2b-e4a2-4fab-ab20-81eb92486337","Type":"ContainerStarted","Data":"e2c6b9fdb8cc22131a87c89d72a9d383bbee2d9de011837d4f3dc13a8b60785e"}
Oct 30 04:15:32.223354 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:32.223243    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6abe4c2b-e4a2-4fab-ab20-81eb92486337","Type":"ContainerStarted","Data":"0a4e116dcabdced0282940cf4aa9d650758e949d144265f1c605b1797b28376b"}
Oct 30 04:15:34.395012 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:34.394984    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:34.426102 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:34.426078    4960 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:34.495826 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:34.495801    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:34.495940 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:34.495902    4960 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:34.511129 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:34.511108    4960 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") DevicePath \"csi-6f8c8e4c1326657251c30594036033694cadf25e97408e4f9d9da436f78f3fa4\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:34.613585 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:34.613546    4960 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/8c516923ea0339579ebf808cef81b9e0aaf55e5b619e2c6187025fa3b2b51f7f/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:34.640985 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:34.640954    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:34.713481 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:34.713426    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:35.029909 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:35.029883    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:15:35.031202 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:15:35.031158    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc289ddb6_01c2_458e_ad4f_59dd07861347.slice/crio-5417f167d173b2310d429308d436b2cfc45ec949c6b8296de5f3605d5007375a WatchSource:0}: Error finding container 5417f167d173b2310d429308d436b2cfc45ec949c6b8296de5f3605d5007375a: Status 404 returned error can't find the container with id 5417f167d173b2310d429308d436b2cfc45ec949c6b8296de5f3605d5007375a
Oct 30 04:15:35.238413 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:35.238386    4960 generic.go:356] "Generic (PLEG): container finished" podID="6abe4c2b-e4a2-4fab-ab20-81eb92486337" containerID="e2c6b9fdb8cc22131a87c89d72a9d383bbee2d9de011837d4f3dc13a8b60785e" exitCode=0
Oct 30 04:15:35.238531 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:35.238464    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6abe4c2b-e4a2-4fab-ab20-81eb92486337","Type":"ContainerDied","Data":"e2c6b9fdb8cc22131a87c89d72a9d383bbee2d9de011837d4f3dc13a8b60785e"}
Oct 30 04:15:35.239828 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:35.239806    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c289ddb6-01c2-458e-ad4f-59dd07861347","Type":"ContainerStarted","Data":"5417f167d173b2310d429308d436b2cfc45ec949c6b8296de5f3605d5007375a"}
Oct 30 04:15:36.252701 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:36.252662    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6abe4c2b-e4a2-4fab-ab20-81eb92486337","Type":"ContainerStarted","Data":"f64e79e18d4730d1be0f01e1ff2c38eb2aa802cc04c7d4aa82421c21ee80929c"}
Oct 30 04:15:36.253122 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:36.252748    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6abe4c2b-e4a2-4fab-ab20-81eb92486337","Type":"ContainerStarted","Data":"91dbb1f0a9ccde2480b3a4bb17a21bec75137cff57ba39867d2570a968d4ab73"}
Oct 30 04:15:36.253122 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:36.252950    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:36.254347 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:36.254323    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c289ddb6-01c2-458e-ad4f-59dd07861347","Type":"ContainerStarted","Data":"4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b"}
Oct 30 04:15:36.323606 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:36.323562    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=2.837202698 podStartE2EDuration="6.323550499s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.486347801s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:36.309548787 +0000 UTC m=+5616.958597207" watchObservedRunningTime="2026-10-30 04:15:36.323550499 +0000 UTC m=+5616.972598920"
Oct 30 04:15:38.193973 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:38.193944    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"
Oct 30 04:15:39.271158 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:39.271075    4960 generic.go:356] "Generic (PLEG): container finished" podID="c289ddb6-01c2-458e-ad4f-59dd07861347" containerID="4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b" exitCode=0
Oct 30 04:15:39.271158 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:39.271121    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c289ddb6-01c2-458e-ad4f-59dd07861347","Type":"ContainerDied","Data":"4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b"}
Oct 30 04:15:40.278479 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:40.278446    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c289ddb6-01c2-458e-ad4f-59dd07861347","Type":"ContainerStarted","Data":"6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc"}
Oct 30 04:15:40.278843 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:40.278511    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c289ddb6-01c2-458e-ad4f-59dd07861347","Type":"ContainerStarted","Data":"8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3"}
Oct 30 04:15:40.278843 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:40.278536    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c289ddb6-01c2-458e-ad4f-59dd07861347","Type":"ContainerStarted","Data":"b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32"}
Oct 30 04:15:40.368029 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:40.367982    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=6.983664923 podStartE2EDuration="10.367969084s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.384304161s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:40.353459669 +0000 UTC m=+5621.002508089" watchObservedRunningTime="2026-10-30 04:15:40.367969084 +0000 UTC m=+5621.017017505"
Oct 30 04:15:42.260059 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:42.260028    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:15:44.713482 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:44.713452    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:46.117482 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:46.117445    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:46.223030 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:46.222995    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:46.340081 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:46.340048    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:47.784023 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:47.783988    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:15:47.784471 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:47.784203    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="d958dfa3-1c20-4ac1-b00a-b67cecf0ee63" containerName="otc-container" containerID="cri-o://d7a038355120261abce1634bd6c5cffc4ee54bdcd2134793e1c2d131e45e53ba" gracePeriod=30
Oct 30 04:15:47.917679 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:47.917651    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:48.039289 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.039173    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:48.051735 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.051709    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.101995 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.101966    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-kube-api-access-n7dzw\" (UniqueName: \"kubernetes.io/projected/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-kube-api-access-n7dzw\") pod \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") "
Oct 30 04:15:48.102117 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.102075    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-otlp-tls-certs\") pod \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") "
Oct 30 04:15:48.102198 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.102117    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-data-science-collector-ta-client-cert\") pod \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") "
Oct 30 04:15:48.102198 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.102165    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-tls-certs\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-tls-certs\") pod \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") "
Oct 30 04:15:48.102328 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.102243    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-otc-internal\") pod \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") "
Oct 30 04:15:48.102328 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.102293    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-monitoring-tls-certs\") pod \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\" (UID: \"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63\") "
Oct 30 04:15:48.103393 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.103361    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-otc-internal" pod "d958dfa3-1c20-4ac1-b00a-b67cecf0ee63" (UID: "d958dfa3-1c20-4ac1-b00a-b67cecf0ee63"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:48.104773 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.104733    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-kube-api-access-n7dzw" pod "d958dfa3-1c20-4ac1-b00a-b67cecf0ee63" (UID: "d958dfa3-1c20-4ac1-b00a-b67cecf0ee63"). InnerVolumeSpecName "kube-api-access-n7dzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:48.104943 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.104921    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-data-science-collector-ta-client-cert" pod "d958dfa3-1c20-4ac1-b00a-b67cecf0ee63" (UID: "d958dfa3-1c20-4ac1-b00a-b67cecf0ee63"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:48.105066 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.105045    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-monitoring-tls-certs" pod "d958dfa3-1c20-4ac1-b00a-b67cecf0ee63" (UID: "d958dfa3-1c20-4ac1-b00a-b67cecf0ee63"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:48.105431 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.105410    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-otlp-tls-certs" pod "d958dfa3-1c20-4ac1-b00a-b67cecf0ee63" (UID: "d958dfa3-1c20-4ac1-b00a-b67cecf0ee63"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:48.105687 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.105662    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-tls-certs" pod "d958dfa3-1c20-4ac1-b00a-b67cecf0ee63" (UID: "d958dfa3-1c20-4ac1-b00a-b67cecf0ee63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:48.203551 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.203515    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:48.203551 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.203547    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:48.203551 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.203557    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:48.203769 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.203566    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7dzw\" (UniqueName: \"kubernetes.io/projected/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-kube-api-access-n7dzw\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:48.203769 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.203574    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:48.203769 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.203583    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:15:48.247802 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.247774    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:48.321037 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.321005    4960 generic.go:356] "Generic (PLEG): container finished" podID="d958dfa3-1c20-4ac1-b00a-b67cecf0ee63" containerID="d7a038355120261abce1634bd6c5cffc4ee54bdcd2134793e1c2d131e45e53ba" exitCode=0
Oct 30 04:15:48.321208 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.321053    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63","Type":"ContainerDied","Data":"d7a038355120261abce1634bd6c5cffc4ee54bdcd2134793e1c2d131e45e53ba"}
Oct 30 04:15:48.321208 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.321099    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d958dfa3-1c20-4ac1-b00a-b67cecf0ee63","Type":"ContainerDied","Data":"44778e39910d519dbe9dade9682ea2f32fcf1501382d57d6baa527a39a01e84e"}
Oct 30 04:15:48.321208 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.321058    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.321208 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.321128    4960 scope.go:122] "RemoveContainer" containerID="d7a038355120261abce1634bd6c5cffc4ee54bdcd2134793e1c2d131e45e53ba"
Oct 30 04:15:48.330094 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.330076    4960 scope.go:122] "RemoveContainer" containerID="d7a038355120261abce1634bd6c5cffc4ee54bdcd2134793e1c2d131e45e53ba"
Oct 30 04:15:48.330365 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:15:48.330344    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7a038355120261abce1634bd6c5cffc4ee54bdcd2134793e1c2d131e45e53ba\": container with ID starting with d7a038355120261abce1634bd6c5cffc4ee54bdcd2134793e1c2d131e45e53ba not found: ID does not exist" containerID="d7a038355120261abce1634bd6c5cffc4ee54bdcd2134793e1c2d131e45e53ba"
Oct 30 04:15:48.330421 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.330378    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7a038355120261abce1634bd6c5cffc4ee54bdcd2134793e1c2d131e45e53ba"} err="failed to get container status \"d7a038355120261abce1634bd6c5cffc4ee54bdcd2134793e1c2d131e45e53ba\": rpc error: code = NotFound desc = could not find container \"d7a038355120261abce1634bd6c5cffc4ee54bdcd2134793e1c2d131e45e53ba\": container with ID starting with d7a038355120261abce1634bd6c5cffc4ee54bdcd2134793e1c2d131e45e53ba not found: ID does not exist"
Oct 30 04:15:48.406741 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.406708    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:15:48.419963 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.419944    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:15:48.444261 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.444236    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:15:48.444620 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.444608    4960 cpu_manager.go:414] "containerMap: removing container" podUID="d958dfa3-1c20-4ac1-b00a-b67cecf0ee63" containerName="otc-container"
Oct 30 04:15:48.444663 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.444622    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d958dfa3-1c20-4ac1-b00a-b67cecf0ee63" containerName="otc-container"
Oct 30 04:15:48.444698 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.444686    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="d958dfa3-1c20-4ac1-b00a-b67cecf0ee63" containerName="otc-container"
Oct 30 04:15:48.448895 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.448881    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.466643 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.466604    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:15:48.466772 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.466641    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-m9vgz\""
Oct 30 04:15:48.466772 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.466642    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:15:48.466772 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.466680    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:15:48.466772 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.466688    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:15:48.466976 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.466931    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:15:48.486277 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.486252    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:15:48.506456 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.506432    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2da56956-79ac-4f27-ae4e-7cc492b1c836-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.506551 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.506471    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.506551 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.506494    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.506664 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.506646    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.506702 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.506675    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.506749 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.506709    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgjm7\" (UniqueName: \"kubernetes.io/projected/2da56956-79ac-4f27-ae4e-7cc492b1c836-kube-api-access-tgjm7\") pod \"data-science-collector-collector-1\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.607463 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.607374    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2da56956-79ac-4f27-ae4e-7cc492b1c836-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.607463 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.607456    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.607705 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.607491    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.607705 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.607659    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.607705 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.607696    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.607865 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.607753    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgjm7\" (UniqueName: \"kubernetes.io/projected/2da56956-79ac-4f27-ae4e-7cc492b1c836-kube-api-access-tgjm7\") pod \"data-science-collector-collector-1\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.608250 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.608227    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2da56956-79ac-4f27-ae4e-7cc492b1c836-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.610919 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.610332    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.610919 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.610640    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.610919 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.610858    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.610919 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.610890    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.627650 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.627627    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgjm7\" (UniqueName: \"kubernetes.io/projected/2da56956-79ac-4f27-ae4e-7cc492b1c836-kube-api-access-tgjm7\") pod \"data-science-collector-collector-1\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:48.760678 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:48.760653    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:15:49.052670 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:49.052638    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:15:49.053716 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:15:49.053694    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da56956_79ac_4f27_ae4e_7cc492b1c836.slice/crio-6da13406593868b900980397a391a86717bcbbb75b77a5f4bec5dedf26980d80 WatchSource:0}: Error finding container 6da13406593868b900980397a391a86717bcbbb75b77a5f4bec5dedf26980d80: Status 404 returned error can't find the container with id 6da13406593868b900980397a391a86717bcbbb75b77a5f4bec5dedf26980d80
Oct 30 04:15:49.055974 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:49.055935    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:15:49.056060 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:49.056004    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:15:49.328823 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:49.328794    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2da56956-79ac-4f27-ae4e-7cc492b1c836","Type":"ContainerStarted","Data":"61e409bcb21bf281820ee5c57676a330e0fcee5df1879499cc0eb612cabd8bd0"}
Oct 30 04:15:49.328980 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:49.328844    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2da56956-79ac-4f27-ae4e-7cc492b1c836","Type":"ContainerStarted","Data":"6da13406593868b900980397a391a86717bcbbb75b77a5f4bec5dedf26980d80"}
Oct 30 04:15:49.372006 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:49.371959    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.3719457259999999 podStartE2EDuration="1.371945726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:49.370607997 +0000 UTC m=+5630.019656420" watchObservedRunningTime="2026-10-30 04:15:49.371945726 +0000 UTC m=+5630.020994147"
Oct 30 04:15:49.502129 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:49.502096    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d958dfa3-1c20-4ac1-b00a-b67cecf0ee63" path="/var/lib/kubelet/pods/d958dfa3-1c20-4ac1-b00a-b67cecf0ee63/volumes"
Oct 30 04:15:49.714206 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:49.714109    4960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:49.719567 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:49.719550    4960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:50.338612 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:50.338585    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:53.953567 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:15:53.953523    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:15:57.306457 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:57.306422    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:57.417236 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:57.417194    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:57.544306 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:57.544272    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:57.654487 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:57.654454    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:58.995557 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:58.995163    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:15:58.998905 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:58.996215    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="2da56956-79ac-4f27-ae4e-7cc492b1c836" containerName="otc-container" containerID="cri-o://61e409bcb21bf281820ee5c57676a330e0fcee5df1879499cc0eb612cabd8bd0" gracePeriod=30
Oct 30 04:15:59.118283 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:59.118252    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:59.236559 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:59.236529    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:59.352900 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:15:59.352863    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:00.253108 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.253085    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:00.300954 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.300910    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:00.379324 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.379241    4960 generic.go:356] "Generic (PLEG): container finished" podID="2da56956-79ac-4f27-ae4e-7cc492b1c836" containerID="61e409bcb21bf281820ee5c57676a330e0fcee5df1879499cc0eb612cabd8bd0" exitCode=0
Oct 30 04:16:00.379324 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.379298    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:00.379537 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.379330    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2da56956-79ac-4f27-ae4e-7cc492b1c836","Type":"ContainerDied","Data":"61e409bcb21bf281820ee5c57676a330e0fcee5df1879499cc0eb612cabd8bd0"}
Oct 30 04:16:00.379537 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.379398    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2da56956-79ac-4f27-ae4e-7cc492b1c836","Type":"ContainerDied","Data":"6da13406593868b900980397a391a86717bcbbb75b77a5f4bec5dedf26980d80"}
Oct 30 04:16:00.379537 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.379432    4960 scope.go:122] "RemoveContainer" containerID="61e409bcb21bf281820ee5c57676a330e0fcee5df1879499cc0eb612cabd8bd0"
Oct 30 04:16:00.388224 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.388202    4960 scope.go:122] "RemoveContainer" containerID="61e409bcb21bf281820ee5c57676a330e0fcee5df1879499cc0eb612cabd8bd0"
Oct 30 04:16:00.388501 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:16:00.388483    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61e409bcb21bf281820ee5c57676a330e0fcee5df1879499cc0eb612cabd8bd0\": container with ID starting with 61e409bcb21bf281820ee5c57676a330e0fcee5df1879499cc0eb612cabd8bd0 not found: ID does not exist" containerID="61e409bcb21bf281820ee5c57676a330e0fcee5df1879499cc0eb612cabd8bd0"
Oct 30 04:16:00.388564 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.388509    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61e409bcb21bf281820ee5c57676a330e0fcee5df1879499cc0eb612cabd8bd0"} err="failed to get container status \"61e409bcb21bf281820ee5c57676a330e0fcee5df1879499cc0eb612cabd8bd0\": rpc error: code = NotFound desc = could not find container \"61e409bcb21bf281820ee5c57676a330e0fcee5df1879499cc0eb612cabd8bd0\": container with ID starting with 61e409bcb21bf281820ee5c57676a330e0fcee5df1879499cc0eb612cabd8bd0 not found: ID does not exist"
Oct 30 04:16:00.407892 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.407732    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-tls-certs\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-tls-certs\") pod \"2da56956-79ac-4f27-ae4e-7cc492b1c836\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") "
Oct 30 04:16:00.407892 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.407757    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da56956-79ac-4f27-ae4e-7cc492b1c836-kube-api-access-tgjm7\" (UniqueName: \"kubernetes.io/projected/2da56956-79ac-4f27-ae4e-7cc492b1c836-kube-api-access-tgjm7\") pod \"2da56956-79ac-4f27-ae4e-7cc492b1c836\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") "
Oct 30 04:16:00.407892 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.407796    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-data-science-collector-ta-client-cert\") pod \"2da56956-79ac-4f27-ae4e-7cc492b1c836\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") "
Oct 30 04:16:00.408077 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.407910    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da56956-79ac-4f27-ae4e-7cc492b1c836-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2da56956-79ac-4f27-ae4e-7cc492b1c836-otc-internal\") pod \"2da56956-79ac-4f27-ae4e-7cc492b1c836\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") "
Oct 30 04:16:00.408077 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.407998    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-otlp-tls-certs\") pod \"2da56956-79ac-4f27-ae4e-7cc492b1c836\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") "
Oct 30 04:16:00.408077 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.408037    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-monitoring-tls-certs\") pod \"2da56956-79ac-4f27-ae4e-7cc492b1c836\" (UID: \"2da56956-79ac-4f27-ae4e-7cc492b1c836\") "
Oct 30 04:16:00.408412 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.408357    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da56956-79ac-4f27-ae4e-7cc492b1c836-otc-internal" pod "2da56956-79ac-4f27-ae4e-7cc492b1c836" (UID: "2da56956-79ac-4f27-ae4e-7cc492b1c836"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:00.410712 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.410686    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-data-science-collector-ta-client-cert" pod "2da56956-79ac-4f27-ae4e-7cc492b1c836" (UID: "2da56956-79ac-4f27-ae4e-7cc492b1c836"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:00.411138 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.410812    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-otlp-tls-certs" pod "2da56956-79ac-4f27-ae4e-7cc492b1c836" (UID: "2da56956-79ac-4f27-ae4e-7cc492b1c836"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:00.411138 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.410863    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da56956-79ac-4f27-ae4e-7cc492b1c836-kube-api-access-tgjm7" pod "2da56956-79ac-4f27-ae4e-7cc492b1c836" (UID: "2da56956-79ac-4f27-ae4e-7cc492b1c836"). InnerVolumeSpecName "kube-api-access-tgjm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:00.411138 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.411044    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-monitoring-tls-certs" pod "2da56956-79ac-4f27-ae4e-7cc492b1c836" (UID: "2da56956-79ac-4f27-ae4e-7cc492b1c836"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:00.425317 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.425288    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-tls-certs" pod "2da56956-79ac-4f27-ae4e-7cc492b1c836" (UID: "2da56956-79ac-4f27-ae4e-7cc492b1c836"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:00.508831 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.508807    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:00.508831 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.508831    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2da56956-79ac-4f27-ae4e-7cc492b1c836-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:00.508977 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.508841    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:00.508977 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.508849    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:00.508977 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.508858    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da56956-79ac-4f27-ae4e-7cc492b1c836-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:00.508977 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.508866    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgjm7\" (UniqueName: \"kubernetes.io/projected/2da56956-79ac-4f27-ae4e-7cc492b1c836-kube-api-access-tgjm7\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:00.723912 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.723883    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:00.737606 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.737574    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:00.760520 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.760497    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:00.760871 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.760858    4960 cpu_manager.go:414] "containerMap: removing container" podUID="2da56956-79ac-4f27-ae4e-7cc492b1c836" containerName="otc-container"
Oct 30 04:16:00.760914 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.760873    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da56956-79ac-4f27-ae4e-7cc492b1c836" containerName="otc-container"
Oct 30 04:16:00.760949 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.760943    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da56956-79ac-4f27-ae4e-7cc492b1c836" containerName="otc-container"
Oct 30 04:16:00.765673 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.765655    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:00.782961 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.782313    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:16:00.782961 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.782679    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:16:00.782961 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.782700    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-m9vgz\""
Oct 30 04:16:00.782961 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.782785    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:16:00.782961 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.782855    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:16:00.786295 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.786268    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:00.795732 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.795717    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:16:00.912892 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.912866    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:00.913003 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.912897    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqdps\" (UniqueName: \"kubernetes.io/projected/2c916e2e-d626-473c-b114-f5cd0560222a-kube-api-access-bqdps\") pod \"data-science-collector-collector-1\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:00.913003 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.912928    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:00.913076 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.913027    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:00.913076 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.913058    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2c916e2e-d626-473c-b114-f5cd0560222a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:00.913143 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:00.913088    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:01.014398 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:01.014337    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2c916e2e-d626-473c-b114-f5cd0560222a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:01.014398 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:01.014376    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:01.014573 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:01.014408    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:01.014573 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:01.014517    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqdps\" (UniqueName: \"kubernetes.io/projected/2c916e2e-d626-473c-b114-f5cd0560222a-kube-api-access-bqdps\") pod \"data-science-collector-collector-1\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:01.014573 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:01.014564    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:01.014758 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:01.014603    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:01.015144 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:01.015122    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2c916e2e-d626-473c-b114-f5cd0560222a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:01.017097 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:01.017077    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:01.017208 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:01.017154    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:01.017318 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:01.017301    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:01.017410 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:01.017395    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:01.035609 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:01.035589    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqdps\" (UniqueName: \"kubernetes.io/projected/2c916e2e-d626-473c-b114-f5cd0560222a-kube-api-access-bqdps\") pod \"data-science-collector-collector-1\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:01.078801 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:01.078783    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:01.369935 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:01.369911    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:01.371230 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:16:01.371201    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c916e2e_d626_473c_b114_f5cd0560222a.slice/crio-b365c2a44386d09bbb6d6f224475673953cd70be121302a8989ce9a2ebc2c030 WatchSource:0}: Error finding container b365c2a44386d09bbb6d6f224475673953cd70be121302a8989ce9a2ebc2c030: Status 404 returned error can't find the container with id b365c2a44386d09bbb6d6f224475673953cd70be121302a8989ce9a2ebc2c030
Oct 30 04:16:01.373811 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:01.373781    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:16:01.373912 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:01.373846    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:16:01.386093 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:01.386071    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2c916e2e-d626-473c-b114-f5cd0560222a","Type":"ContainerStarted","Data":"b365c2a44386d09bbb6d6f224475673953cd70be121302a8989ce9a2ebc2c030"}
Oct 30 04:16:01.509072 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:01.509039    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da56956-79ac-4f27-ae4e-7cc492b1c836" path="/var/lib/kubelet/pods/2da56956-79ac-4f27-ae4e-7cc492b1c836/volumes"
Oct 30 04:16:02.392713 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:02.392674    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2c916e2e-d626-473c-b114-f5cd0560222a","Type":"ContainerStarted","Data":"2434facc175df9fdcff3eaf7e5d3d8c72c3060d094cb95e815691dc45c427437"}
Oct 30 04:16:02.435960 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:02.435911    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.43589888 podStartE2EDuration="2.43589888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:02.434310186 +0000 UTC m=+5643.083358608" watchObservedRunningTime="2026-10-30 04:16:02.43589888 +0000 UTC m=+5643.084947300"
Oct 30 04:16:03.276214 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.276164    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"]
Oct 30 04:16:03.276393 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.276369    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj" podUID="87d89dab-459a-499a-b417-7079799bcf56" containerName="korrel8r" containerID="cri-o://83073ec0201f8179f21f0fc796a209df04a94aa6a7ce9b127a5689138dadc186" gracePeriod=30
Oct 30 04:16:03.398587 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.398552    4960 generic.go:356] "Generic (PLEG): container finished" podID="87d89dab-459a-499a-b417-7079799bcf56" containerID="83073ec0201f8179f21f0fc796a209df04a94aa6a7ce9b127a5689138dadc186" exitCode=0
Oct 30 04:16:03.398943 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.398617    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj" event={"ID":"87d89dab-459a-499a-b417-7079799bcf56","Type":"ContainerDied","Data":"83073ec0201f8179f21f0fc796a209df04a94aa6a7ce9b127a5689138dadc186"}
Oct 30 04:16:03.524545 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.524521    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"
Oct 30 04:16:03.632211 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.632158    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:03.637817 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.637790    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87d89dab-459a-499a-b417-7079799bcf56-kube-api-access-n7rpf\" (UniqueName: \"kubernetes.io/projected/87d89dab-459a-499a-b417-7079799bcf56-kube-api-access-n7rpf\") pod \"87d89dab-459a-499a-b417-7079799bcf56\" (UID: \"87d89dab-459a-499a-b417-7079799bcf56\") "
Oct 30 04:16:03.638069 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.638049    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87d89dab-459a-499a-b417-7079799bcf56-tls\" (UniqueName: \"kubernetes.io/secret/87d89dab-459a-499a-b417-7079799bcf56-tls\") pod \"87d89dab-459a-499a-b417-7079799bcf56\" (UID: \"87d89dab-459a-499a-b417-7079799bcf56\") "
Oct 30 04:16:03.638222 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.638207    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87d89dab-459a-499a-b417-7079799bcf56-config\" (UniqueName: \"kubernetes.io/configmap/87d89dab-459a-499a-b417-7079799bcf56-config\") pod \"87d89dab-459a-499a-b417-7079799bcf56\" (UID: \"87d89dab-459a-499a-b417-7079799bcf56\") "
Oct 30 04:16:03.638952 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.638922    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d89dab-459a-499a-b417-7079799bcf56-config" pod "87d89dab-459a-499a-b417-7079799bcf56" (UID: "87d89dab-459a-499a-b417-7079799bcf56"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:03.642541 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.642512    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87d89dab-459a-499a-b417-7079799bcf56-tls" pod "87d89dab-459a-499a-b417-7079799bcf56" (UID: "87d89dab-459a-499a-b417-7079799bcf56"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:03.643760 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.643730    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d89dab-459a-499a-b417-7079799bcf56-kube-api-access-n7rpf" pod "87d89dab-459a-499a-b417-7079799bcf56" (UID: "87d89dab-459a-499a-b417-7079799bcf56"). InnerVolumeSpecName "kube-api-access-n7rpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:03.705944 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.705907    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:16:03.706306 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.706275    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="c289ddb6-01c2-458e-ad4f-59dd07861347" containerName="prometheus" containerID="cri-o://b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32" gracePeriod=600
Oct 30 04:16:03.706413 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.706293    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="c289ddb6-01c2-458e-ad4f-59dd07861347" containerName="thanos-sidecar" containerID="cri-o://6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc" gracePeriod=600
Oct 30 04:16:03.706413 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.706326    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="c289ddb6-01c2-458e-ad4f-59dd07861347" containerName="config-reloader" containerID="cri-o://8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3" gracePeriod=600
Oct 30 04:16:03.713169 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.713144    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:16:03.713473 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.713446    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6abe4c2b-e4a2-4fab-ab20-81eb92486337" containerName="alertmanager" containerID="cri-o://91dbb1f0a9ccde2480b3a4bb17a21bec75137cff57ba39867d2570a968d4ab73" gracePeriod=120
Oct 30 04:16:03.713577 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.713528    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6abe4c2b-e4a2-4fab-ab20-81eb92486337" containerName="config-reloader" containerID="cri-o://f64e79e18d4730d1be0f01e1ff2c38eb2aa802cc04c7d4aa82421c21ee80929c" gracePeriod=120
Oct 30 04:16:03.739511 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.739490    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7rpf\" (UniqueName: \"kubernetes.io/projected/87d89dab-459a-499a-b417-7079799bcf56-kube-api-access-n7rpf\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:03.739511 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.739513    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/87d89dab-459a-499a-b417-7079799bcf56-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:03.739641 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.739523    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/87d89dab-459a-499a-b417-7079799bcf56-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:03.760194 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.760157    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:03.873205 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.873156    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:03.973149 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:03.973125    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:04.142754 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.142712    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c289ddb6-01c2-458e-ad4f-59dd07861347-tls-assets\" (UniqueName: \"kubernetes.io/projected/c289ddb6-01c2-458e-ad4f-59dd07861347-tls-assets\") pod \"c289ddb6-01c2-458e-ad4f-59dd07861347\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") "
Oct 30 04:16:04.142754 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.142758    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"c289ddb6-01c2-458e-ad4f-59dd07861347\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") "
Oct 30 04:16:04.142982 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.142870    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"c289ddb6-01c2-458e-ad4f-59dd07861347\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") "
Oct 30 04:16:04.142982 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.142905    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"c289ddb6-01c2-458e-ad4f-59dd07861347\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") "
Oct 30 04:16:04.142982 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.142954    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-secret-prometheus-web-tls-ca\") pod \"c289ddb6-01c2-458e-ad4f-59dd07861347\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") "
Oct 30 04:16:04.143153 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.142991    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"c289ddb6-01c2-458e-ad4f-59dd07861347\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") "
Oct 30 04:16:04.143153 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.143044    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"c289ddb6-01c2-458e-ad4f-59dd07861347\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") "
Oct 30 04:16:04.143153 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.143076    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"c289ddb6-01c2-458e-ad4f-59dd07861347\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") "
Oct 30 04:16:04.143153 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.143108    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-config\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-config\") pod \"c289ddb6-01c2-458e-ad4f-59dd07861347\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") "
Oct 30 04:16:04.143408 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.143211    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config\") pod \"c289ddb6-01c2-458e-ad4f-59dd07861347\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") "
Oct 30 04:16:04.143408 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.143255    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c289ddb6-01c2-458e-ad4f-59dd07861347-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c289ddb6-01c2-458e-ad4f-59dd07861347-config-out\") pod \"c289ddb6-01c2-458e-ad4f-59dd07861347\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") "
Oct 30 04:16:04.143408 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.143308    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c289ddb6-01c2-458e-ad4f-59dd07861347-kube-api-access-n784q\" (UniqueName: \"kubernetes.io/projected/c289ddb6-01c2-458e-ad4f-59dd07861347-kube-api-access-n784q\") pod \"c289ddb6-01c2-458e-ad4f-59dd07861347\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") "
Oct 30 04:16:04.143408 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.143344    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-thanos-prometheus-http-client-file\") pod \"c289ddb6-01c2-458e-ad4f-59dd07861347\" (UID: \"c289ddb6-01c2-458e-ad4f-59dd07861347\") "
Oct 30 04:16:04.143937 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.143810    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-0" pod "c289ddb6-01c2-458e-ad4f-59dd07861347" (UID: "c289ddb6-01c2-458e-ad4f-59dd07861347"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:04.143937 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.143897    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-1" pod "c289ddb6-01c2-458e-ad4f-59dd07861347" (UID: "c289ddb6-01c2-458e-ad4f-59dd07861347"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:04.144522 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.144229    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-2" pod "c289ddb6-01c2-458e-ad4f-59dd07861347" (UID: "c289ddb6-01c2-458e-ad4f-59dd07861347"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:04.146380 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.146353    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "c289ddb6-01c2-458e-ad4f-59dd07861347" (UID: "c289ddb6-01c2-458e-ad4f-59dd07861347"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:04.146380 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.146363    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c289ddb6-01c2-458e-ad4f-59dd07861347-tls-assets" pod "c289ddb6-01c2-458e-ad4f-59dd07861347" (UID: "c289ddb6-01c2-458e-ad4f-59dd07861347"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:04.146555 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.146444    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-secret-prometheus-web-tls-ca" pod "c289ddb6-01c2-458e-ad4f-59dd07861347" (UID: "c289ddb6-01c2-458e-ad4f-59dd07861347"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:04.146555 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.146462    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "c289ddb6-01c2-458e-ad4f-59dd07861347" (UID: "c289ddb6-01c2-458e-ad4f-59dd07861347"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:04.147549 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.147523    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-thanos-prometheus-http-client-file" pod "c289ddb6-01c2-458e-ad4f-59dd07861347" (UID: "c289ddb6-01c2-458e-ad4f-59dd07861347"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:04.147649 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.147543    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-config" pod "c289ddb6-01c2-458e-ad4f-59dd07861347" (UID: "c289ddb6-01c2-458e-ad4f-59dd07861347"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:04.148737 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.148712    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c289ddb6-01c2-458e-ad4f-59dd07861347-kube-api-access-n784q" pod "c289ddb6-01c2-458e-ad4f-59dd07861347" (UID: "c289ddb6-01c2-458e-ad4f-59dd07861347"). InnerVolumeSpecName "kube-api-access-n784q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:04.148935 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.148906    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c289ddb6-01c2-458e-ad4f-59dd07861347-config-out" pod "c289ddb6-01c2-458e-ad4f-59dd07861347" (UID: "c289ddb6-01c2-458e-ad4f-59dd07861347"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:04.149849 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.149826    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87" pod "c289ddb6-01c2-458e-ad4f-59dd07861347" (UID: "c289ddb6-01c2-458e-ad4f-59dd07861347"). InnerVolumeSpecName "pvc-9689c530-7f91-4584-b28e-8809bed9e799". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:16:04.159591 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.159565    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config" pod "c289ddb6-01c2-458e-ad4f-59dd07861347" (UID: "c289ddb6-01c2-458e-ad4f-59dd07861347"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:04.244216 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.244171    4960 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c289ddb6-01c2-458e-ad4f-59dd07861347-tls-assets\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.244216 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.244215    4960 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.244438 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.244244    4960 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") on node \"ip-10-0-128-241.ec2.internal\" "
Oct 30 04:16:04.244438 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.244256    4960 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.244438 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.244267    4960 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-secret-prometheus-web-tls-ca\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.244438 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.244291    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.244438 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.244301    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.244438 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.244311    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c289ddb6-01c2-458e-ad4f-59dd07861347-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.244438 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.244319    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.244438 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.244327    4960 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-web-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.244438 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.244335    4960 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c289ddb6-01c2-458e-ad4f-59dd07861347-config-out\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.244438 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.244343    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n784q\" (UniqueName: \"kubernetes.io/projected/c289ddb6-01c2-458e-ad4f-59dd07861347-kube-api-access-n784q\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.244438 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.244351    4960 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c289ddb6-01c2-458e-ad4f-59dd07861347-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.255773 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.255754    4960 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-9689c530-7f91-4584-b28e-8809bed9e799" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87") on node "ip-10-0-128-241.ec2.internal"
Oct 30 04:16:04.339956 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.339921    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:04.345792 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.345760    4960 reconciler_common.go:299] "Volume detached for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"csi-6f8c8e4c1326657251c30594036033694cadf25e97408e4f9d9da436f78f3fa4\""
Oct 30 04:16:04.406997 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.406916    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:04.407899 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.407874    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj" event={"ID":"87d89dab-459a-499a-b417-7079799bcf56","Type":"ContainerDied","Data":"ae20aed180515f0dc3a65b2b56d4f285ea2b3cde01a1efdcd1f9b597e1c4e80e"}
Oct 30 04:16:04.408006 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.407926    4960 scope.go:122] "RemoveContainer" containerID="83073ec0201f8179f21f0fc796a209df04a94aa6a7ce9b127a5689138dadc186"
Oct 30 04:16:04.408065 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.408030    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"
Oct 30 04:16:04.415053 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.415024    4960 generic.go:356] "Generic (PLEG): container finished" podID="6abe4c2b-e4a2-4fab-ab20-81eb92486337" containerID="f64e79e18d4730d1be0f01e1ff2c38eb2aa802cc04c7d4aa82421c21ee80929c" exitCode=0
Oct 30 04:16:04.415229 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.415117    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6abe4c2b-e4a2-4fab-ab20-81eb92486337","Type":"ContainerDied","Data":"f64e79e18d4730d1be0f01e1ff2c38eb2aa802cc04c7d4aa82421c21ee80929c"}
Oct 30 04:16:04.418160 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.418130    4960 generic.go:356] "Generic (PLEG): container finished" podID="c289ddb6-01c2-458e-ad4f-59dd07861347" containerID="6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc" exitCode=0
Oct 30 04:16:04.418398 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.418163    4960 generic.go:356] "Generic (PLEG): container finished" podID="c289ddb6-01c2-458e-ad4f-59dd07861347" containerID="8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3" exitCode=0
Oct 30 04:16:04.418398 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.418192    4960 generic.go:356] "Generic (PLEG): container finished" podID="c289ddb6-01c2-458e-ad4f-59dd07861347" containerID="b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32" exitCode=0
Oct 30 04:16:04.418589 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.418501    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="2c916e2e-d626-473c-b114-f5cd0560222a" containerName="otc-container" containerID="cri-o://2434facc175df9fdcff3eaf7e5d3d8c72c3060d094cb95e815691dc45c427437" gracePeriod=30
Oct 30 04:16:04.418589 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.418520    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:04.418858 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.418819    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c289ddb6-01c2-458e-ad4f-59dd07861347","Type":"ContainerDied","Data":"6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc"}
Oct 30 04:16:04.419643 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.419606    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c289ddb6-01c2-458e-ad4f-59dd07861347","Type":"ContainerDied","Data":"8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3"}
Oct 30 04:16:04.420323 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.419785    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c289ddb6-01c2-458e-ad4f-59dd07861347","Type":"ContainerDied","Data":"b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32"}
Oct 30 04:16:04.421105 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.420835    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c289ddb6-01c2-458e-ad4f-59dd07861347","Type":"ContainerDied","Data":"5417f167d173b2310d429308d436b2cfc45ec949c6b8296de5f3605d5007375a"}
Oct 30 04:16:04.430578 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.430546    4960 scope.go:122] "RemoveContainer" containerID="6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc"
Oct 30 04:16:04.445804 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.445773    4960 scope.go:122] "RemoveContainer" containerID="8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3"
Oct 30 04:16:04.526481 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.526448    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:04.528307 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.528279    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:16:04.532538 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.532517    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:16:04.534115 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.534092    4960 scope.go:122] "RemoveContainer" containerID="b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32"
Oct 30 04:16:04.542888 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.542867    4960 scope.go:122] "RemoveContainer" containerID="4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b"
Oct 30 04:16:04.553247 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.553228    4960 scope.go:122] "RemoveContainer" containerID="6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc"
Oct 30 04:16:04.553575 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:16:04.553549    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc\": container with ID starting with 6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc not found: ID does not exist" containerID="6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc"
Oct 30 04:16:04.553671 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.553587    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc"} err="failed to get container status \"6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc\": rpc error: code = NotFound desc = could not find container \"6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc\": container with ID starting with 6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc not found: ID does not exist"
Oct 30 04:16:04.553671 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.553616    4960 scope.go:122] "RemoveContainer" containerID="8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3"
Oct 30 04:16:04.553926 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:16:04.553905    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3\": container with ID starting with 8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3 not found: ID does not exist" containerID="8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3"
Oct 30 04:16:04.554010 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.553940    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3"} err="failed to get container status \"8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3\": rpc error: code = NotFound desc = could not find container \"8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3\": container with ID starting with 8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3 not found: ID does not exist"
Oct 30 04:16:04.554010 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.553964    4960 scope.go:122] "RemoveContainer" containerID="b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32"
Oct 30 04:16:04.554364 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:16:04.554337    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32\": container with ID starting with b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32 not found: ID does not exist" containerID="b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32"
Oct 30 04:16:04.554442 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.554371    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32"} err="failed to get container status \"b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32\": rpc error: code = NotFound desc = could not find container \"b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32\": container with ID starting with b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32 not found: ID does not exist"
Oct 30 04:16:04.554442 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.554391    4960 scope.go:122] "RemoveContainer" containerID="4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b"
Oct 30 04:16:04.554653 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:16:04.554636    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b\": container with ID starting with 4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b not found: ID does not exist" containerID="4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b"
Oct 30 04:16:04.554713 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.554658    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b"} err="failed to get container status \"4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b\": rpc error: code = NotFound desc = could not find container \"4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b\": container with ID starting with 4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b not found: ID does not exist"
Oct 30 04:16:04.554713 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.554670    4960 scope.go:122] "RemoveContainer" containerID="6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc"
Oct 30 04:16:04.554972 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.554942    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc"} err="failed to get container status \"6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc\": rpc error: code = NotFound desc = could not find container \"6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc\": container with ID starting with 6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc not found: ID does not exist"
Oct 30 04:16:04.555039 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.554973    4960 scope.go:122] "RemoveContainer" containerID="8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3"
Oct 30 04:16:04.555289 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.555259    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3"} err="failed to get container status \"8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3\": rpc error: code = NotFound desc = could not find container \"8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3\": container with ID starting with 8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3 not found: ID does not exist"
Oct 30 04:16:04.555289 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.555288    4960 scope.go:122] "RemoveContainer" containerID="b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32"
Oct 30 04:16:04.555568 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.555543    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32"} err="failed to get container status \"b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32\": rpc error: code = NotFound desc = could not find container \"b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32\": container with ID starting with b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32 not found: ID does not exist"
Oct 30 04:16:04.555568 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.555568    4960 scope.go:122] "RemoveContainer" containerID="4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b"
Oct 30 04:16:04.555834 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.555814    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b"} err="failed to get container status \"4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b\": rpc error: code = NotFound desc = could not find container \"4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b\": container with ID starting with 4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b not found: ID does not exist"
Oct 30 04:16:04.555918 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.555834    4960 scope.go:122] "RemoveContainer" containerID="6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc"
Oct 30 04:16:04.556190 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.556150    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc"} err="failed to get container status \"6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc\": rpc error: code = NotFound desc = could not find container \"6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc\": container with ID starting with 6ce02124375b5095af202c37cead38dfe91d1ef3de574675301215f044d98afc not found: ID does not exist"
Oct 30 04:16:04.556269 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.556174    4960 scope.go:122] "RemoveContainer" containerID="8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3"
Oct 30 04:16:04.556492 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.556460    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3"} err="failed to get container status \"8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3\": rpc error: code = NotFound desc = could not find container \"8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3\": container with ID starting with 8ede80672a9251a9d3567bd15914a180156be120d5b967e2aea38ccad6624ad3 not found: ID does not exist"
Oct 30 04:16:04.556492 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.556492    4960 scope.go:122] "RemoveContainer" containerID="b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32"
Oct 30 04:16:04.556752 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.556733    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32"} err="failed to get container status \"b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32\": rpc error: code = NotFound desc = could not find container \"b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32\": container with ID starting with b4a930fa3620ec05a3d1c3655d0875c5281f1ceefe949c02932e637997219e32 not found: ID does not exist"
Oct 30 04:16:04.556807 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.556752    4960 scope.go:122] "RemoveContainer" containerID="4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b"
Oct 30 04:16:04.556996 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.556980    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b"} err="failed to get container status \"4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b\": rpc error: code = NotFound desc = could not find container \"4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b\": container with ID starting with 4d6dae9ce7e48152d83e1eb9e806c0bcd7449305c264be2e3a261f154cdc9f3b not found: ID does not exist"
Oct 30 04:16:04.569941 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.569914    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"]
Oct 30 04:16:04.573714 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.573693    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-nm8rj"]
Oct 30 04:16:04.665906 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.665829    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:04.696791 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.696771    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:04.765521 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.765494    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-data-science-collector-ta-client-cert\") pod \"2c916e2e-d626-473c-b114-f5cd0560222a\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") "
Oct 30 04:16:04.767794 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.767768    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-data-science-collector-ta-client-cert" pod "2c916e2e-d626-473c-b114-f5cd0560222a" (UID: "2c916e2e-d626-473c-b114-f5cd0560222a"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:04.865963 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.865935    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-monitoring-tls-certs\") pod \"2c916e2e-d626-473c-b114-f5cd0560222a\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") "
Oct 30 04:16:04.866099 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.865976    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c916e2e-d626-473c-b114-f5cd0560222a-kube-api-access-bqdps\" (UniqueName: \"kubernetes.io/projected/2c916e2e-d626-473c-b114-f5cd0560222a-kube-api-access-bqdps\") pod \"2c916e2e-d626-473c-b114-f5cd0560222a\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") "
Oct 30 04:16:04.866099 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.865998    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c916e2e-d626-473c-b114-f5cd0560222a-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2c916e2e-d626-473c-b114-f5cd0560222a-otc-internal\") pod \"2c916e2e-d626-473c-b114-f5cd0560222a\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") "
Oct 30 04:16:04.866099 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.866027    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-tls-certs\") pod \"2c916e2e-d626-473c-b114-f5cd0560222a\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") "
Oct 30 04:16:04.866099 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.866085    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-otlp-tls-certs\") pod \"2c916e2e-d626-473c-b114-f5cd0560222a\" (UID: \"2c916e2e-d626-473c-b114-f5cd0560222a\") "
Oct 30 04:16:04.866370 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.866309    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.866430 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.866364    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c916e2e-d626-473c-b114-f5cd0560222a-otc-internal" pod "2c916e2e-d626-473c-b114-f5cd0560222a" (UID: "2c916e2e-d626-473c-b114-f5cd0560222a"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:04.868249 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.868225    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-tls-certs" pod "2c916e2e-d626-473c-b114-f5cd0560222a" (UID: "2c916e2e-d626-473c-b114-f5cd0560222a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:04.868348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.868248    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-monitoring-tls-certs" pod "2c916e2e-d626-473c-b114-f5cd0560222a" (UID: "2c916e2e-d626-473c-b114-f5cd0560222a"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:04.868348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.868294    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c916e2e-d626-473c-b114-f5cd0560222a-kube-api-access-bqdps" pod "2c916e2e-d626-473c-b114-f5cd0560222a" (UID: "2c916e2e-d626-473c-b114-f5cd0560222a"). InnerVolumeSpecName "kube-api-access-bqdps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:04.868348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.868339    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-otlp-tls-certs" pod "2c916e2e-d626-473c-b114-f5cd0560222a" (UID: "2c916e2e-d626-473c-b114-f5cd0560222a"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:04.966635 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.966581    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.966635 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.966603    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.966635 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.966612    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqdps\" (UniqueName: \"kubernetes.io/projected/2c916e2e-d626-473c-b114-f5cd0560222a-kube-api-access-bqdps\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.966635 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.966621    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2c916e2e-d626-473c-b114-f5cd0560222a-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.966635 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:04.966630    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c916e2e-d626-473c-b114-f5cd0560222a-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:05.321719 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:05.321662    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:05.429233 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:05.429192    4960 generic.go:356] "Generic (PLEG): container finished" podID="2c916e2e-d626-473c-b114-f5cd0560222a" containerID="2434facc175df9fdcff3eaf7e5d3d8c72c3060d094cb95e815691dc45c427437" exitCode=0
Oct 30 04:16:05.429727 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:05.429257    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:05.429727 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:05.429273    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2c916e2e-d626-473c-b114-f5cd0560222a","Type":"ContainerDied","Data":"2434facc175df9fdcff3eaf7e5d3d8c72c3060d094cb95e815691dc45c427437"}
Oct 30 04:16:05.429727 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:05.429327    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2c916e2e-d626-473c-b114-f5cd0560222a","Type":"ContainerDied","Data":"b365c2a44386d09bbb6d6f224475673953cd70be121302a8989ce9a2ebc2c030"}
Oct 30 04:16:05.429727 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:05.429354    4960 scope.go:122] "RemoveContainer" containerID="2434facc175df9fdcff3eaf7e5d3d8c72c3060d094cb95e815691dc45c427437"
Oct 30 04:16:05.432060 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:05.432035    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:05.440336 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:05.440313    4960 scope.go:122] "RemoveContainer" containerID="2434facc175df9fdcff3eaf7e5d3d8c72c3060d094cb95e815691dc45c427437"
Oct 30 04:16:05.440588 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:16:05.440571    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2434facc175df9fdcff3eaf7e5d3d8c72c3060d094cb95e815691dc45c427437\": container with ID starting with 2434facc175df9fdcff3eaf7e5d3d8c72c3060d094cb95e815691dc45c427437 not found: ID does not exist" containerID="2434facc175df9fdcff3eaf7e5d3d8c72c3060d094cb95e815691dc45c427437"
Oct 30 04:16:05.440675 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:05.440596    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2434facc175df9fdcff3eaf7e5d3d8c72c3060d094cb95e815691dc45c427437"} err="failed to get container status \"2434facc175df9fdcff3eaf7e5d3d8c72c3060d094cb95e815691dc45c427437\": rpc error: code = NotFound desc = could not find container \"2434facc175df9fdcff3eaf7e5d3d8c72c3060d094cb95e815691dc45c427437\": container with ID starting with 2434facc175df9fdcff3eaf7e5d3d8c72c3060d094cb95e815691dc45c427437 not found: ID does not exist"
Oct 30 04:16:05.485592 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:05.485564    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:05.488775 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:05.488748    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:05.502215 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:05.502162    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c916e2e-d626-473c-b114-f5cd0560222a" path="/var/lib/kubelet/pods/2c916e2e-d626-473c-b114-f5cd0560222a/volumes"
Oct 30 04:16:05.502628 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:05.502610    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87d89dab-459a-499a-b417-7079799bcf56" path="/var/lib/kubelet/pods/87d89dab-459a-499a-b417-7079799bcf56/volumes"
Oct 30 04:16:05.503076 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:05.503058    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c289ddb6-01c2-458e-ad4f-59dd07861347" path="/var/lib/kubelet/pods/c289ddb6-01c2-458e-ad4f-59dd07861347/volumes"
Oct 30 04:16:05.536916 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:05.536890    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:07.259343 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:07.259304    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6abe4c2b-e4a2-4fab-ab20-81eb92486337" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.128:9093/-/ready\": dial tcp 10.132.0.128:9093: connect: connection refused"
Oct 30 04:16:09.768130 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:09.768101    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:09.876493 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:09.876465    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:09.983811 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:09.983758    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:10.111478 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.111446    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:10.173911 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.173884    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"]
Oct 30 04:16:10.174291 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.174273    4960 cpu_manager.go:414] "containerMap: removing container" podUID="c289ddb6-01c2-458e-ad4f-59dd07861347" containerName="thanos-sidecar"
Oct 30 04:16:10.174393 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.174295    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c289ddb6-01c2-458e-ad4f-59dd07861347" containerName="thanos-sidecar"
Oct 30 04:16:10.174393 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.174321    4960 cpu_manager.go:414] "containerMap: removing container" podUID="c289ddb6-01c2-458e-ad4f-59dd07861347" containerName="config-reloader"
Oct 30 04:16:10.174393 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.174329    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c289ddb6-01c2-458e-ad4f-59dd07861347" containerName="config-reloader"
Oct 30 04:16:10.174393 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.174346    4960 cpu_manager.go:414] "containerMap: removing container" podUID="c289ddb6-01c2-458e-ad4f-59dd07861347" containerName="init-config-reloader"
Oct 30 04:16:10.174393 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.174355    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c289ddb6-01c2-458e-ad4f-59dd07861347" containerName="init-config-reloader"
Oct 30 04:16:10.174393 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.174373    4960 cpu_manager.go:414] "containerMap: removing container" podUID="c289ddb6-01c2-458e-ad4f-59dd07861347" containerName="prometheus"
Oct 30 04:16:10.174393 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.174381    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c289ddb6-01c2-458e-ad4f-59dd07861347" containerName="prometheus"
Oct 30 04:16:10.174741 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.174399    4960 cpu_manager.go:414] "containerMap: removing container" podUID="87d89dab-459a-499a-b417-7079799bcf56" containerName="korrel8r"
Oct 30 04:16:10.174741 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.174406    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87d89dab-459a-499a-b417-7079799bcf56" containerName="korrel8r"
Oct 30 04:16:10.174741 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.174426    4960 cpu_manager.go:414] "containerMap: removing container" podUID="2c916e2e-d626-473c-b114-f5cd0560222a" containerName="otc-container"
Oct 30 04:16:10.174741 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.174433    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c916e2e-d626-473c-b114-f5cd0560222a" containerName="otc-container"
Oct 30 04:16:10.174741 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.174518    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="87d89dab-459a-499a-b417-7079799bcf56" containerName="korrel8r"
Oct 30 04:16:10.174741 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.174532    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c916e2e-d626-473c-b114-f5cd0560222a" containerName="otc-container"
Oct 30 04:16:10.174741 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.174544    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="c289ddb6-01c2-458e-ad4f-59dd07861347" containerName="thanos-sidecar"
Oct 30 04:16:10.174741 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.174555    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="c289ddb6-01c2-458e-ad4f-59dd07861347" containerName="config-reloader"
Oct 30 04:16:10.174741 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.174567    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="c289ddb6-01c2-458e-ad4f-59dd07861347" containerName="prometheus"
Oct 30 04:16:10.179117 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.179097    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"
Oct 30 04:16:10.194955 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.194933    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"]
Oct 30 04:16:10.201765 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.201744    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q56x2\" (UniqueName: \"kubernetes.io/projected/b38749d2-d835-4cf1-970f-faa3453dac62-kube-api-access-q56x2\") pod \"korrel8r-5cbcc89b8f-rn2fq\" (UID: \"b38749d2-d835-4cf1-970f-faa3453dac62\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"
Oct 30 04:16:10.201922 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.201898    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b38749d2-d835-4cf1-970f-faa3453dac62-config\") pod \"korrel8r-5cbcc89b8f-rn2fq\" (UID: \"b38749d2-d835-4cf1-970f-faa3453dac62\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"
Oct 30 04:16:10.201972 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.201946    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b38749d2-d835-4cf1-970f-faa3453dac62-tls\") pod \"korrel8r-5cbcc89b8f-rn2fq\" (UID: \"b38749d2-d835-4cf1-970f-faa3453dac62\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"
Oct 30 04:16:10.207536 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.207519    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-4vh2g\""
Oct 30 04:16:10.207642 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.207629    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\""
Oct 30 04:16:10.218896 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.218867    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:10.271303 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.271275    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\""
Oct 30 04:16:10.302760 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.302732    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q56x2\" (UniqueName: \"kubernetes.io/projected/b38749d2-d835-4cf1-970f-faa3453dac62-kube-api-access-q56x2\") pod \"korrel8r-5cbcc89b8f-rn2fq\" (UID: \"b38749d2-d835-4cf1-970f-faa3453dac62\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"
Oct 30 04:16:10.302857 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.302826    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b38749d2-d835-4cf1-970f-faa3453dac62-config\") pod \"korrel8r-5cbcc89b8f-rn2fq\" (UID: \"b38749d2-d835-4cf1-970f-faa3453dac62\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"
Oct 30 04:16:10.302898 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.302857    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b38749d2-d835-4cf1-970f-faa3453dac62-tls\") pod \"korrel8r-5cbcc89b8f-rn2fq\" (UID: \"b38749d2-d835-4cf1-970f-faa3453dac62\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"
Oct 30 04:16:10.303562 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.303536    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b38749d2-d835-4cf1-970f-faa3453dac62-config\") pod \"korrel8r-5cbcc89b8f-rn2fq\" (UID: \"b38749d2-d835-4cf1-970f-faa3453dac62\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"
Oct 30 04:16:10.305487 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.305467    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b38749d2-d835-4cf1-970f-faa3453dac62-tls\") pod \"korrel8r-5cbcc89b8f-rn2fq\" (UID: \"b38749d2-d835-4cf1-970f-faa3453dac62\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"
Oct 30 04:16:10.323174 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.323149    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q56x2\" (UniqueName: \"kubernetes.io/projected/b38749d2-d835-4cf1-970f-faa3453dac62-kube-api-access-q56x2\") pod \"korrel8r-5cbcc89b8f-rn2fq\" (UID: \"b38749d2-d835-4cf1-970f-faa3453dac62\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"
Oct 30 04:16:10.424228 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.424143    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:10.491438 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.491409    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"
Oct 30 04:16:10.688006 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.687931    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:10.781519 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.781494    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"]
Oct 30 04:16:10.784219 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:16:10.784168    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38749d2_d835_4cf1_970f_faa3453dac62.slice/crio-32c51451a338482ba74d559419da8c7c60faa35a6ec62ab695cad11508d18aab WatchSource:0}: Error finding container 32c51451a338482ba74d559419da8c7c60faa35a6ec62ab695cad11508d18aab: Status 404 returned error can't find the container with id 32c51451a338482ba74d559419da8c7c60faa35a6ec62ab695cad11508d18aab
Oct 30 04:16:10.823614 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:10.823592    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:11.024741 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.024664    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:11.457208 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.457150    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq" event={"ID":"b38749d2-d835-4cf1-970f-faa3453dac62","Type":"ContainerStarted","Data":"b4e41647ccb0054b710a55305ac9d667b2f7ca060216013aad4fb208e2c17199"}
Oct 30 04:16:11.457392 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.457240    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq" event={"ID":"b38749d2-d835-4cf1-970f-faa3453dac62","Type":"ContainerStarted","Data":"32c51451a338482ba74d559419da8c7c60faa35a6ec62ab695cad11508d18aab"}
Oct 30 04:16:11.457392 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.457293    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"
Oct 30 04:16:11.500879 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.500825    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq" podStartSLOduration=1.500807878 podStartE2EDuration="1.500807878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:11.498944209 +0000 UTC m=+5652.147992631" watchObservedRunningTime="2026-10-30 04:16:11.500807878 +0000 UTC m=+5652.149856303"
Oct 30 04:16:11.555869 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.555836    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:11.559404 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.559376    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:11.578477 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.578446    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:11.608548 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.608398    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:16:11.608950 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.608905    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:11.615841 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.615806    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:11.615988 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.615890    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:11.615988 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.615925    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:11.615988 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.615964    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:11.616210 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.615988    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8lft\" (UniqueName: \"kubernetes.io/projected/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-kube-api-access-j8lft\") pod \"data-science-collector-collector-1\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:11.616210 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.616059    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:11.643547 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.643522    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:16:11.643680 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.643650    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:16:11.643680 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.643657    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8rxn8\""
Oct 30 04:16:11.643680 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.643661    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:16:11.643807 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.643684    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:16:11.717285 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.717053    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:11.717285 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.717087    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8lft\" (UniqueName: \"kubernetes.io/projected/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-kube-api-access-j8lft\") pod \"data-science-collector-collector-1\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:11.717285 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.717130    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:11.717285 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.717168    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:11.717285 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.717240    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:11.717285 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.717264    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:11.717884 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.717851    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:11.719890 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.719863    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:11.719987 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.719970    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:11.720097 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.720075    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:11.720135 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.720079    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:11.740013 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.739983    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:11.740956 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.740936    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8lft\" (UniqueName: \"kubernetes.io/projected/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-kube-api-access-j8lft\") pod \"data-science-collector-collector-1\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:11.848285 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.848256    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:11.876209 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:11.876167    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:12.169738 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:12.169715    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:12.170954 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:16:12.170927    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1016b0e4_3ce8_49f2_91ad_9d60d32a3505.slice/crio-347839c8c3a94bb6c61a8201fd94db4dd1e289e0d50d9436fffe852ce0ce0955 WatchSource:0}: Error finding container 347839c8c3a94bb6c61a8201fd94db4dd1e289e0d50d9436fffe852ce0ce0955: Status 404 returned error can't find the container with id 347839c8c3a94bb6c61a8201fd94db4dd1e289e0d50d9436fffe852ce0ce0955
Oct 30 04:16:12.173065 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:12.173037    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:16:12.173131 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:12.173092    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:16:12.259664 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:12.259625    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6abe4c2b-e4a2-4fab-ab20-81eb92486337" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.128:9093/-/ready\": dial tcp 10.132.0.128:9093: connect: connection refused"
Oct 30 04:16:12.464859 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:12.464785    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1016b0e4-3ce8-49f2-91ad-9d60d32a3505","Type":"ContainerStarted","Data":"3a405f96b545d150726a595ea6dd7584a7baefdc4694e2a34bf2f40a3fab7480"}
Oct 30 04:16:12.464859 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:12.464838    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1016b0e4-3ce8-49f2-91ad-9d60d32a3505","Type":"ContainerStarted","Data":"347839c8c3a94bb6c61a8201fd94db4dd1e289e0d50d9436fffe852ce0ce0955"}
Oct 30 04:16:12.508545 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:12.508501    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.50848981 podStartE2EDuration="1.50848981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:12.507529045 +0000 UTC m=+5653.156577467" watchObservedRunningTime="2026-10-30 04:16:12.50848981 +0000 UTC m=+5653.157538231"
Oct 30 04:16:13.976661 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:13.976609    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:14.034360 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.034321    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-web-config\" (UniqueName: \"kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-web-config\") pod \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") "
Oct 30 04:16:14.034525 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.034382    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-config-volume\" (UniqueName: \"kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-config-volume\") pod \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") "
Oct 30 04:16:14.034525 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.034416    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abe4c2b-e4a2-4fab-ab20-81eb92486337-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6abe4c2b-e4a2-4fab-ab20-81eb92486337-config-out\") pod \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") "
Oct 30 04:16:14.034525 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.034448    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abe4c2b-e4a2-4fab-ab20-81eb92486337-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6abe4c2b-e4a2-4fab-ab20-81eb92486337-alertmanager-data-science-monitoringstack-db\") pod \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") "
Oct 30 04:16:14.034734 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.034525    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-cluster-tls-config\") pod \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") "
Oct 30 04:16:14.034734 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.034569    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6abe4c2b-e4a2-4fab-ab20-81eb92486337-tls-assets\" (UniqueName: \"kubernetes.io/projected/6abe4c2b-e4a2-4fab-ab20-81eb92486337-tls-assets\") pod \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") "
Oct 30 04:16:14.034734 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.034598    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6abe4c2b-e4a2-4fab-ab20-81eb92486337-kube-api-access-7r6jx\" (UniqueName: \"kubernetes.io/projected/6abe4c2b-e4a2-4fab-ab20-81eb92486337-kube-api-access-7r6jx\") pod \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\" (UID: \"6abe4c2b-e4a2-4fab-ab20-81eb92486337\") "
Oct 30 04:16:14.035463 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.035434    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abe4c2b-e4a2-4fab-ab20-81eb92486337-alertmanager-data-science-monitoringstack-db" pod "6abe4c2b-e4a2-4fab-ab20-81eb92486337" (UID: "6abe4c2b-e4a2-4fab-ab20-81eb92486337"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:14.038471 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.038258    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abe4c2b-e4a2-4fab-ab20-81eb92486337-config-out" pod "6abe4c2b-e4a2-4fab-ab20-81eb92486337" (UID: "6abe4c2b-e4a2-4fab-ab20-81eb92486337"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:14.038891 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.038857    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-config-volume" pod "6abe4c2b-e4a2-4fab-ab20-81eb92486337" (UID: "6abe4c2b-e4a2-4fab-ab20-81eb92486337"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:14.038986 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.038901    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abe4c2b-e4a2-4fab-ab20-81eb92486337-kube-api-access-7r6jx" pod "6abe4c2b-e4a2-4fab-ab20-81eb92486337" (UID: "6abe4c2b-e4a2-4fab-ab20-81eb92486337"). InnerVolumeSpecName "kube-api-access-7r6jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:14.039799 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.039775    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abe4c2b-e4a2-4fab-ab20-81eb92486337-tls-assets" pod "6abe4c2b-e4a2-4fab-ab20-81eb92486337" (UID: "6abe4c2b-e4a2-4fab-ab20-81eb92486337"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:14.043870 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.043843    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-cluster-tls-config" pod "6abe4c2b-e4a2-4fab-ab20-81eb92486337" (UID: "6abe4c2b-e4a2-4fab-ab20-81eb92486337"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:14.052385 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.052364    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-web-config" pod "6abe4c2b-e4a2-4fab-ab20-81eb92486337" (UID: "6abe4c2b-e4a2-4fab-ab20-81eb92486337"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:14.135631 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.135611    4960 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6abe4c2b-e4a2-4fab-ab20-81eb92486337-tls-assets\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:14.135631 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.135630    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r6jx\" (UniqueName: \"kubernetes.io/projected/6abe4c2b-e4a2-4fab-ab20-81eb92486337-kube-api-access-7r6jx\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:14.135757 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.135642    4960 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-web-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:14.135757 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.135651    4960 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-config-volume\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:14.135757 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.135658    4960 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6abe4c2b-e4a2-4fab-ab20-81eb92486337-config-out\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:14.135757 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.135667    4960 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6abe4c2b-e4a2-4fab-ab20-81eb92486337-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:14.135757 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.135675    4960 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6abe4c2b-e4a2-4fab-ab20-81eb92486337-cluster-tls-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:14.419452 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.419422    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:14.476098 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.476063    4960 generic.go:356] "Generic (PLEG): container finished" podID="6abe4c2b-e4a2-4fab-ab20-81eb92486337" containerID="91dbb1f0a9ccde2480b3a4bb17a21bec75137cff57ba39867d2570a968d4ab73" exitCode=0
Oct 30 04:16:14.476278 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.476126    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:14.476278 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.476136    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6abe4c2b-e4a2-4fab-ab20-81eb92486337","Type":"ContainerDied","Data":"91dbb1f0a9ccde2480b3a4bb17a21bec75137cff57ba39867d2570a968d4ab73"}
Oct 30 04:16:14.476278 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.476224    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6abe4c2b-e4a2-4fab-ab20-81eb92486337","Type":"ContainerDied","Data":"0a4e116dcabdced0282940cf4aa9d650758e949d144265f1c605b1797b28376b"}
Oct 30 04:16:14.476278 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.476261    4960 scope.go:122] "RemoveContainer" containerID="f64e79e18d4730d1be0f01e1ff2c38eb2aa802cc04c7d4aa82421c21ee80929c"
Oct 30 04:16:14.486103 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.486080    4960 scope.go:122] "RemoveContainer" containerID="91dbb1f0a9ccde2480b3a4bb17a21bec75137cff57ba39867d2570a968d4ab73"
Oct 30 04:16:14.495821 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.495793    4960 scope.go:122] "RemoveContainer" containerID="e2c6b9fdb8cc22131a87c89d72a9d383bbee2d9de011837d4f3dc13a8b60785e"
Oct 30 04:16:14.504436 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:16:14.504403    4960 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abe4c2b_e4a2_4fab_ab20_81eb92486337.slice/crio-0a4e116dcabdced0282940cf4aa9d650758e949d144265f1c605b1797b28376b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abe4c2b_e4a2_4fab_ab20_81eb92486337.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 04:16:14.504716 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.504695    4960 scope.go:122] "RemoveContainer" containerID="f64e79e18d4730d1be0f01e1ff2c38eb2aa802cc04c7d4aa82421c21ee80929c"
Oct 30 04:16:14.504972 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:16:14.504956    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f64e79e18d4730d1be0f01e1ff2c38eb2aa802cc04c7d4aa82421c21ee80929c\": container with ID starting with f64e79e18d4730d1be0f01e1ff2c38eb2aa802cc04c7d4aa82421c21ee80929c not found: ID does not exist" containerID="f64e79e18d4730d1be0f01e1ff2c38eb2aa802cc04c7d4aa82421c21ee80929c"
Oct 30 04:16:14.505032 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.504981    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f64e79e18d4730d1be0f01e1ff2c38eb2aa802cc04c7d4aa82421c21ee80929c"} err="failed to get container status \"f64e79e18d4730d1be0f01e1ff2c38eb2aa802cc04c7d4aa82421c21ee80929c\": rpc error: code = NotFound desc = could not find container \"f64e79e18d4730d1be0f01e1ff2c38eb2aa802cc04c7d4aa82421c21ee80929c\": container with ID starting with f64e79e18d4730d1be0f01e1ff2c38eb2aa802cc04c7d4aa82421c21ee80929c not found: ID does not exist"
Oct 30 04:16:14.505032 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.505004    4960 scope.go:122] "RemoveContainer" containerID="91dbb1f0a9ccde2480b3a4bb17a21bec75137cff57ba39867d2570a968d4ab73"
Oct 30 04:16:14.505268 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:16:14.505241    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91dbb1f0a9ccde2480b3a4bb17a21bec75137cff57ba39867d2570a968d4ab73\": container with ID starting with 91dbb1f0a9ccde2480b3a4bb17a21bec75137cff57ba39867d2570a968d4ab73 not found: ID does not exist" containerID="91dbb1f0a9ccde2480b3a4bb17a21bec75137cff57ba39867d2570a968d4ab73"
Oct 30 04:16:14.505382 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.505276    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91dbb1f0a9ccde2480b3a4bb17a21bec75137cff57ba39867d2570a968d4ab73"} err="failed to get container status \"91dbb1f0a9ccde2480b3a4bb17a21bec75137cff57ba39867d2570a968d4ab73\": rpc error: code = NotFound desc = could not find container \"91dbb1f0a9ccde2480b3a4bb17a21bec75137cff57ba39867d2570a968d4ab73\": container with ID starting with 91dbb1f0a9ccde2480b3a4bb17a21bec75137cff57ba39867d2570a968d4ab73 not found: ID does not exist"
Oct 30 04:16:14.505382 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.505298    4960 scope.go:122] "RemoveContainer" containerID="e2c6b9fdb8cc22131a87c89d72a9d383bbee2d9de011837d4f3dc13a8b60785e"
Oct 30 04:16:14.505577 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:16:14.505558    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c6b9fdb8cc22131a87c89d72a9d383bbee2d9de011837d4f3dc13a8b60785e\": container with ID starting with e2c6b9fdb8cc22131a87c89d72a9d383bbee2d9de011837d4f3dc13a8b60785e not found: ID does not exist" containerID="e2c6b9fdb8cc22131a87c89d72a9d383bbee2d9de011837d4f3dc13a8b60785e"
Oct 30 04:16:14.505640 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.505587    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c6b9fdb8cc22131a87c89d72a9d383bbee2d9de011837d4f3dc13a8b60785e"} err="failed to get container status \"e2c6b9fdb8cc22131a87c89d72a9d383bbee2d9de011837d4f3dc13a8b60785e\": rpc error: code = NotFound desc = could not find container \"e2c6b9fdb8cc22131a87c89d72a9d383bbee2d9de011837d4f3dc13a8b60785e\": container with ID starting with e2c6b9fdb8cc22131a87c89d72a9d383bbee2d9de011837d4f3dc13a8b60785e not found: ID does not exist"
Oct 30 04:16:14.544391 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.544367    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:16:14.558079 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.558056    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:16:14.658609 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.658581    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:14.761796 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:14.761766    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:15.080768 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.080738    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:16:15.081091 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.081037    4960 cpu_manager.go:414] "containerMap: removing container" podUID="6abe4c2b-e4a2-4fab-ab20-81eb92486337" containerName="alertmanager"
Oct 30 04:16:15.081091 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.081048    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6abe4c2b-e4a2-4fab-ab20-81eb92486337" containerName="alertmanager"
Oct 30 04:16:15.081091 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.081056    4960 cpu_manager.go:414] "containerMap: removing container" podUID="6abe4c2b-e4a2-4fab-ab20-81eb92486337" containerName="config-reloader"
Oct 30 04:16:15.081091 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.081061    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6abe4c2b-e4a2-4fab-ab20-81eb92486337" containerName="config-reloader"
Oct 30 04:16:15.081091 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.081090    4960 cpu_manager.go:414] "containerMap: removing container" podUID="6abe4c2b-e4a2-4fab-ab20-81eb92486337" containerName="init-config-reloader"
Oct 30 04:16:15.081329 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.081095    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6abe4c2b-e4a2-4fab-ab20-81eb92486337" containerName="init-config-reloader"
Oct 30 04:16:15.081329 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.081154    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="6abe4c2b-e4a2-4fab-ab20-81eb92486337" containerName="alertmanager"
Oct 30 04:16:15.081329 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.081161    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="6abe4c2b-e4a2-4fab-ab20-81eb92486337" containerName="config-reloader"
Oct 30 04:16:15.085756 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.085736    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.107150 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.107126    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:16:15.108722 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.108702    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\""
Oct 30 04:16:15.108842 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.108712    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\""
Oct 30 04:16:15.109058 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.109038    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\""
Oct 30 04:16:15.109202 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.109110    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:16:15.109285 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.109224    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-nnncs\""
Oct 30 04:16:15.111961 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.111942    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:15.144630 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.144605    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.144756 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.144657    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.144823 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.144801    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6f6becf4-245d-4106-9b55-680179f120aa-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.144870 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.144841    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bltj5\" (UniqueName: \"kubernetes.io/projected/6f6becf4-245d-4106-9b55-680179f120aa-kube-api-access-bltj5\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.144992 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.144972    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.145059 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.145009    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f6becf4-245d-4106-9b55-680179f120aa-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.145059 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.145036    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f6becf4-245d-4106-9b55-680179f120aa-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.245814 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.245788    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.245909 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.245820    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.245909 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.245847    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6f6becf4-245d-4106-9b55-680179f120aa-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.245909 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.245878    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bltj5\" (UniqueName: \"kubernetes.io/projected/6f6becf4-245d-4106-9b55-680179f120aa-kube-api-access-bltj5\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.246073 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.245915    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.246073 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.245931    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f6becf4-245d-4106-9b55-680179f120aa-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.246073 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.245955    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f6becf4-245d-4106-9b55-680179f120aa-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.246307 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.246281    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6f6becf4-245d-4106-9b55-680179f120aa-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.248545 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.248516    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f6becf4-245d-4106-9b55-680179f120aa-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.248619 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.248560    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f6becf4-245d-4106-9b55-680179f120aa-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.248619 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.248578    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.248919 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.248899    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.249353 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.249335    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.265901 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.265881    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bltj5\" (UniqueName: \"kubernetes.io/projected/6f6becf4-245d-4106-9b55-680179f120aa-kube-api-access-bltj5\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.350172 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.350109    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:16:15.353956 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.353941    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.375479 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.372042    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 04:16:15.375479 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.372510    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 04:16:15.375479 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.372733    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 04:16:15.375479 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.372821    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:16:15.375479 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.372859    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 04:16:15.375479 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.373033    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 04:16:15.375479 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.373348    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-42x42\""
Oct 30 04:16:15.375479 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.373595    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 04:16:15.375479 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.373770    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 04:16:15.375479 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.374136    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:16:15.381000 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.379694    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:16:15.381000 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.380829    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:15.402691 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.402665    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:15.503487 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.503460    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6abe4c2b-e4a2-4fab-ab20-81eb92486337" path="/var/lib/kubelet/pods/6abe4c2b-e4a2-4fab-ab20-81eb92486337/volumes"
Oct 30 04:16:15.549867 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.549812    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.550017 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.549914    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz62l\" (UniqueName: \"kubernetes.io/projected/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-kube-api-access-cz62l\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.550017 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.549982    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.550110 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.550044    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.550151 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.550106    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.550151 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.550133    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.550251 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.550152    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.550293 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.550267    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.550330 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.550297    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.550330 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.550322    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.550407 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.550379    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.550448 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.550411    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.650833 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.650761    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.650833 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.650802    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.651018 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.650838    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.651018 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.650863    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.651018 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.650891    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.651212 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.651062    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.651212 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.651111    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.651212 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.651142    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.651381 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.651222    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.651381 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.651250    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.651381 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.651318    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.651381 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.651348    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz62l\" (UniqueName: \"kubernetes.io/projected/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-kube-api-access-cz62l\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.652246 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.652163    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.652397 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.652373    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.652490 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.652428    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.654285 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.654237    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.654425 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.654405    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.654644 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.654620    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.655131 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.655108    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.655241 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.655140    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.655331 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.655315    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.655776 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.655756    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.656052 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.656035    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.671855 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.671836    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz62l\" (UniqueName: \"kubernetes.io/projected/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-kube-api-access-cz62l\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:15.710890 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:15.710870    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:16:15.712505 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:16:15.712486    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6becf4_245d_4106_9b55_680179f120aa.slice/crio-b46a0f2c90698d2a19975d45960b4d12089e29073130ca30ff5399e6e2b2426e WatchSource:0}: Error finding container b46a0f2c90698d2a19975d45960b4d12089e29073130ca30ff5399e6e2b2426e: Status 404 returned error can't find the container with id b46a0f2c90698d2a19975d45960b4d12089e29073130ca30ff5399e6e2b2426e
Oct 30 04:16:16.487757 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:16.487720    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6f6becf4-245d-4106-9b55-680179f120aa","Type":"ContainerStarted","Data":"c13f3958fe3983d4856457b38683bc01d1184e7fb282a0f8a6973fc65c6dcfdf"}
Oct 30 04:16:16.488117 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:16.487772    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6f6becf4-245d-4106-9b55-680179f120aa","Type":"ContainerStarted","Data":"b46a0f2c90698d2a19975d45960b4d12089e29073130ca30ff5399e6e2b2426e"}
Oct 30 04:16:19.503898 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:19.503820    4960 generic.go:356] "Generic (PLEG): container finished" podID="6f6becf4-245d-4106-9b55-680179f120aa" containerID="c13f3958fe3983d4856457b38683bc01d1184e7fb282a0f8a6973fc65c6dcfdf" exitCode=0
Oct 30 04:16:19.504287 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:19.503902    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6f6becf4-245d-4106-9b55-680179f120aa","Type":"ContainerDied","Data":"c13f3958fe3983d4856457b38683bc01d1184e7fb282a0f8a6973fc65c6dcfdf"}
Oct 30 04:16:20.515290 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:20.515259    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6f6becf4-245d-4106-9b55-680179f120aa","Type":"ContainerStarted","Data":"374fec369e2a0cbaa39b14fc6003afb0dfe70369dddb5cb18d676656e53b1461"}
Oct 30 04:16:20.515653 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:20.515328    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6f6becf4-245d-4106-9b55-680179f120aa","Type":"ContainerStarted","Data":"a1ae8ddaa6a4927c0d7d716a1b407be75e6470aac221998de7c8f837d59fe08a"}
Oct 30 04:16:20.515653 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:20.515542    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:20.587754 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:20.587712    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=2.40220544 podStartE2EDuration="5.587700651s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.185495211s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:20.573752905 +0000 UTC m=+5661.222801326" watchObservedRunningTime="2026-10-30 04:16:20.587700651 +0000 UTC m=+5661.236749105"
Oct 30 04:16:22.466590 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:22.466562    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"
Oct 30 04:16:23.538617 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:23.538577    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:23.618414 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:23.618384    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:23.649097 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:23.649068    4960 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:23.719487 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:23.719461    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:23.719658 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:23.719596    4960 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:23.734964 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:23.734945    4960 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") DevicePath \"csi-6f8c8e4c1326657251c30594036033694cadf25e97408e4f9d9da436f78f3fa4\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:23.840093 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:23.840049    4960 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/8c516923ea0339579ebf808cef81b9e0aaf55e5b619e2c6187025fa3b2b51f7f/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:23.868381 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:23.868363    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:23.953340 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:16:23.953306    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:16:24.081619 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:24.081559    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:24.400782 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:16:24.400723    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cc19dd_d8c0_4c2f_8243_2ca505ccb347.slice/crio-d4ea1135e54580bd7440dd5e8cd880d6fd34d74f1082a61e23ef86571cf4b4b1 WatchSource:0}: Error finding container d4ea1135e54580bd7440dd5e8cd880d6fd34d74f1082a61e23ef86571cf4b4b1: Status 404 returned error can't find the container with id d4ea1135e54580bd7440dd5e8cd880d6fd34d74f1082a61e23ef86571cf4b4b1
Oct 30 04:16:24.413307 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:24.413280    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:16:24.534739 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:24.534708    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"23cc19dd-d8c0-4c2f-8243-2ca505ccb347","Type":"ContainerStarted","Data":"d4ea1135e54580bd7440dd5e8cd880d6fd34d74f1082a61e23ef86571cf4b4b1"}
Oct 30 04:16:25.542801 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:25.542767    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"23cc19dd-d8c0-4c2f-8243-2ca505ccb347","Type":"ContainerStarted","Data":"742d5016609a7cfe13dc8e8e5528f5f83268a68acb4bf3ff4ed1dbfce8e07c43"}
Oct 30 04:16:26.520693 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:26.520664    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:16:28.558756 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:28.558715    4960 generic.go:356] "Generic (PLEG): container finished" podID="23cc19dd-d8c0-4c2f-8243-2ca505ccb347" containerID="742d5016609a7cfe13dc8e8e5528f5f83268a68acb4bf3ff4ed1dbfce8e07c43" exitCode=0
Oct 30 04:16:28.559090 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:28.558779    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"23cc19dd-d8c0-4c2f-8243-2ca505ccb347","Type":"ContainerDied","Data":"742d5016609a7cfe13dc8e8e5528f5f83268a68acb4bf3ff4ed1dbfce8e07c43"}
Oct 30 04:16:29.388984 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:29.388903    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:29.507634 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:29.507606    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:29.568571 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:29.568537    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"23cc19dd-d8c0-4c2f-8243-2ca505ccb347","Type":"ContainerStarted","Data":"116a654479f1e8300adaad8f5a6e1be80c88948373c27a5989d93a691ade1f7f"}
Oct 30 04:16:29.568895 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:29.568597    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"23cc19dd-d8c0-4c2f-8243-2ca505ccb347","Type":"ContainerStarted","Data":"20f1c876437b544a45acd7fd519e4bd6553c40a2db9d6db882e9b81cba2c3c62"}
Oct 30 04:16:29.568895 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:29.568624    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"23cc19dd-d8c0-4c2f-8243-2ca505ccb347","Type":"ContainerStarted","Data":"3828f949825df825e934600ff2971dd1fa6a0af2573547a9d5b2f390675c608e"}
Oct 30 04:16:29.632490 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:29.632459    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:29.642995 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:29.642923    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=11.457665129 podStartE2EDuration="14.642911903s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.185246774s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:29.629157274 +0000 UTC m=+5670.278205695" watchObservedRunningTime="2026-10-30 04:16:29.642911903 +0000 UTC m=+5670.291960324"
Oct 30 04:16:29.744495 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:29.744458    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:31.064493 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.064459    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:31.065427 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.065369    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="1016b0e4-3ce8-49f2-91ad-9d60d32a3505" containerName="otc-container" containerID="cri-o://3a405f96b545d150726a595ea6dd7584a7baefdc4694e2a34bf2f40a3fab7480" gracePeriod=30
Oct 30 04:16:31.196711 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.196671    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:31.309565 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.309486    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:31.330675 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.330653    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.412339 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.412307    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:31.481128 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.481102    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-tls-certs\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-tls-certs\") pod \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") "
Oct 30 04:16:31.481271 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.481162    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-otlp-tls-certs\") pod \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") "
Oct 30 04:16:31.481328 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.481291    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-monitoring-tls-certs\") pod \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") "
Oct 30 04:16:31.481328 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.481320    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-otc-internal\" (UniqueName: \"kubernetes.io/configmap/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-otc-internal\") pod \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") "
Oct 30 04:16:31.481436 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.481347    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-data-science-collector-ta-client-cert\") pod \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") "
Oct 30 04:16:31.481541 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.481495    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-kube-api-access-j8lft\" (UniqueName: \"kubernetes.io/projected/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-kube-api-access-j8lft\") pod \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\" (UID: \"1016b0e4-3ce8-49f2-91ad-9d60d32a3505\") "
Oct 30 04:16:31.481716 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.481691    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-otc-internal" pod "1016b0e4-3ce8-49f2-91ad-9d60d32a3505" (UID: "1016b0e4-3ce8-49f2-91ad-9d60d32a3505"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:31.481844 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.481825    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:31.483539 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.483513    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-tls-certs" pod "1016b0e4-3ce8-49f2-91ad-9d60d32a3505" (UID: "1016b0e4-3ce8-49f2-91ad-9d60d32a3505"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:31.483642 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.483545    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-monitoring-tls-certs" pod "1016b0e4-3ce8-49f2-91ad-9d60d32a3505" (UID: "1016b0e4-3ce8-49f2-91ad-9d60d32a3505"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:31.483771 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.483754    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-kube-api-access-j8lft" pod "1016b0e4-3ce8-49f2-91ad-9d60d32a3505" (UID: "1016b0e4-3ce8-49f2-91ad-9d60d32a3505"). InnerVolumeSpecName "kube-api-access-j8lft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:31.483875 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.483862    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-otlp-tls-certs" pod "1016b0e4-3ce8-49f2-91ad-9d60d32a3505" (UID: "1016b0e4-3ce8-49f2-91ad-9d60d32a3505"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:31.484037 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.484023    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-data-science-collector-ta-client-cert" pod "1016b0e4-3ce8-49f2-91ad-9d60d32a3505" (UID: "1016b0e4-3ce8-49f2-91ad-9d60d32a3505"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:31.578946 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.578916    4960 generic.go:356] "Generic (PLEG): container finished" podID="1016b0e4-3ce8-49f2-91ad-9d60d32a3505" containerID="3a405f96b545d150726a595ea6dd7584a7baefdc4694e2a34bf2f40a3fab7480" exitCode=0
Oct 30 04:16:31.579070 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.578971    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.579070 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.578990    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1016b0e4-3ce8-49f2-91ad-9d60d32a3505","Type":"ContainerDied","Data":"3a405f96b545d150726a595ea6dd7584a7baefdc4694e2a34bf2f40a3fab7480"}
Oct 30 04:16:31.579070 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.579028    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1016b0e4-3ce8-49f2-91ad-9d60d32a3505","Type":"ContainerDied","Data":"347839c8c3a94bb6c61a8201fd94db4dd1e289e0d50d9436fffe852ce0ce0955"}
Oct 30 04:16:31.579070 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.579048    4960 scope.go:122] "RemoveContainer" containerID="3a405f96b545d150726a595ea6dd7584a7baefdc4694e2a34bf2f40a3fab7480"
Oct 30 04:16:31.582527 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.582484    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:31.582527 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.582505    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:31.582527 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.582516    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8lft\" (UniqueName: \"kubernetes.io/projected/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-kube-api-access-j8lft\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:31.582527 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.582526    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:31.582527 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.582534    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1016b0e4-3ce8-49f2-91ad-9d60d32a3505-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:31.587338 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.587319    4960 scope.go:122] "RemoveContainer" containerID="3a405f96b545d150726a595ea6dd7584a7baefdc4694e2a34bf2f40a3fab7480"
Oct 30 04:16:31.587614 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:16:31.587593    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a405f96b545d150726a595ea6dd7584a7baefdc4694e2a34bf2f40a3fab7480\": container with ID starting with 3a405f96b545d150726a595ea6dd7584a7baefdc4694e2a34bf2f40a3fab7480 not found: ID does not exist" containerID="3a405f96b545d150726a595ea6dd7584a7baefdc4694e2a34bf2f40a3fab7480"
Oct 30 04:16:31.587685 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.587621    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a405f96b545d150726a595ea6dd7584a7baefdc4694e2a34bf2f40a3fab7480"} err="failed to get container status \"3a405f96b545d150726a595ea6dd7584a7baefdc4694e2a34bf2f40a3fab7480\": rpc error: code = NotFound desc = could not find container \"3a405f96b545d150726a595ea6dd7584a7baefdc4694e2a34bf2f40a3fab7480\": container with ID starting with 3a405f96b545d150726a595ea6dd7584a7baefdc4694e2a34bf2f40a3fab7480 not found: ID does not exist"
Oct 30 04:16:31.634628 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.634609    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:31.648341 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.648323    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:31.670954 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.670930    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:31.671315 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.671302    4960 cpu_manager.go:414] "containerMap: removing container" podUID="1016b0e4-3ce8-49f2-91ad-9d60d32a3505" containerName="otc-container"
Oct 30 04:16:31.671368 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.671317    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1016b0e4-3ce8-49f2-91ad-9d60d32a3505" containerName="otc-container"
Oct 30 04:16:31.671404 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.671374    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="1016b0e4-3ce8-49f2-91ad-9d60d32a3505" containerName="otc-container"
Oct 30 04:16:31.675690 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.675672    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.692852 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.692829    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:16:31.693044 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.693025    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:16:31.693044 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.693132    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:16:31.693334 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.693228    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:16:31.693334 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.693297    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8rxn8\""
Oct 30 04:16:31.694908 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.694886    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:31.782116 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.782092    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:16:31.784391 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.784371    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.784477 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.784409    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.784572 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.784528    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.784617 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.784574    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q9mg\" (UniqueName: \"kubernetes.io/projected/771b14c6-d117-4db0-85d5-7eecd48c833a-kube-api-access-6q9mg\") pod \"data-science-collector-collector-1\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.784655 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.784629    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/771b14c6-d117-4db0-85d5-7eecd48c833a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.784699 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.784662    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.885276 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.885219    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.885276 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.885262    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.885439 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.885293    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q9mg\" (UniqueName: \"kubernetes.io/projected/771b14c6-d117-4db0-85d5-7eecd48c833a-kube-api-access-6q9mg\") pod \"data-science-collector-collector-1\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.885439 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.885340    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/771b14c6-d117-4db0-85d5-7eecd48c833a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.885439 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.885364    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.885439 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.885416    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.886287 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.886264    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/771b14c6-d117-4db0-85d5-7eecd48c833a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.887930 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.887905    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.888024 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.887964    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.888024 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.888011    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.888105 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.888022    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.905325 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.905306    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q9mg\" (UniqueName: \"kubernetes.io/projected/771b14c6-d117-4db0-85d5-7eecd48c833a-kube-api-access-6q9mg\") pod \"data-science-collector-collector-1\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:31.987570 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:31.987549    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:32.278581 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:32.278555    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:32.280512 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:16:32.280479    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771b14c6_d117_4db0_85d5_7eecd48c833a.slice/crio-d7cb2c66b698f21eed70a9f29a825bd0d75cf5750584a52c81f4e3c3a1620334 WatchSource:0}: Error finding container d7cb2c66b698f21eed70a9f29a825bd0d75cf5750584a52c81f4e3c3a1620334: Status 404 returned error can't find the container with id d7cb2c66b698f21eed70a9f29a825bd0d75cf5750584a52c81f4e3c3a1620334
Oct 30 04:16:32.282973 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:32.282938    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:16:32.283041 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:32.283008    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:16:32.586058 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:32.586024    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"771b14c6-d117-4db0-85d5-7eecd48c833a","Type":"ContainerStarted","Data":"cfc981edf9841516030abe14e46e27b7faf14f48b1ac2421d3244582902658d6"}
Oct 30 04:16:32.586286 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:32.586074    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"771b14c6-d117-4db0-85d5-7eecd48c833a","Type":"ContainerStarted","Data":"d7cb2c66b698f21eed70a9f29a825bd0d75cf5750584a52c81f4e3c3a1620334"}
Oct 30 04:16:32.629244 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:32.629173    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.629157195 podStartE2EDuration="1.629157195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:32.62832057 +0000 UTC m=+5673.277369042" watchObservedRunningTime="2026-10-30 04:16:32.629157195 +0000 UTC m=+5673.278205616"
Oct 30 04:16:33.502491 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:33.502458    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1016b0e4-3ce8-49f2-91ad-9d60d32a3505" path="/var/lib/kubelet/pods/1016b0e4-3ce8-49f2-91ad-9d60d32a3505/volumes"
Oct 30 04:16:34.082217 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:34.082169    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:39.082543 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:39.082512    4960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:39.088010 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:39.087988    4960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:39.623295 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:39.623269    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:46.184137 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:46.184102    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:46.293956 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:46.293926    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:46.414167 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:46.414137    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:46.615657 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:46.615626    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:47.876881 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:47.876849    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:47.877285 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:47.877018    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="771b14c6-d117-4db0-85d5-7eecd48c833a" containerName="otc-container" containerID="cri-o://cfc981edf9841516030abe14e46e27b7faf14f48b1ac2421d3244582902658d6" gracePeriod=30
Oct 30 04:16:48.006000 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:48.005959    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:48.119936 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:48.119908    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:48.226486 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:48.226412    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:50.221997 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.221973    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:50.327943 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.327913    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/771b14c6-d117-4db0-85d5-7eecd48c833a-otc-internal\" (UniqueName: \"kubernetes.io/configmap/771b14c6-d117-4db0-85d5-7eecd48c833a-otc-internal\") pod \"771b14c6-d117-4db0-85d5-7eecd48c833a\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") "
Oct 30 04:16:50.328088 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.327968    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-monitoring-tls-certs\") pod \"771b14c6-d117-4db0-85d5-7eecd48c833a\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") "
Oct 30 04:16:50.328088 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.328022    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-tls-certs\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-tls-certs\") pod \"771b14c6-d117-4db0-85d5-7eecd48c833a\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") "
Oct 30 04:16:50.328088 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.328072    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-data-science-collector-ta-client-cert\") pod \"771b14c6-d117-4db0-85d5-7eecd48c833a\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") "
Oct 30 04:16:50.328299 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.328100    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/771b14c6-d117-4db0-85d5-7eecd48c833a-kube-api-access-6q9mg\" (UniqueName: \"kubernetes.io/projected/771b14c6-d117-4db0-85d5-7eecd48c833a-kube-api-access-6q9mg\") pod \"771b14c6-d117-4db0-85d5-7eecd48c833a\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") "
Oct 30 04:16:50.328299 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.328169    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-otlp-tls-certs\") pod \"771b14c6-d117-4db0-85d5-7eecd48c833a\" (UID: \"771b14c6-d117-4db0-85d5-7eecd48c833a\") "
Oct 30 04:16:50.328413 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.328360    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/771b14c6-d117-4db0-85d5-7eecd48c833a-otc-internal" pod "771b14c6-d117-4db0-85d5-7eecd48c833a" (UID: "771b14c6-d117-4db0-85d5-7eecd48c833a"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:50.328484 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.328462    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/771b14c6-d117-4db0-85d5-7eecd48c833a-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:50.330535 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.330503    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-tls-certs" pod "771b14c6-d117-4db0-85d5-7eecd48c833a" (UID: "771b14c6-d117-4db0-85d5-7eecd48c833a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:50.330662 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.330632    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-data-science-collector-ta-client-cert" pod "771b14c6-d117-4db0-85d5-7eecd48c833a" (UID: "771b14c6-d117-4db0-85d5-7eecd48c833a"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:50.330733 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.330703    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771b14c6-d117-4db0-85d5-7eecd48c833a-kube-api-access-6q9mg" pod "771b14c6-d117-4db0-85d5-7eecd48c833a" (UID: "771b14c6-d117-4db0-85d5-7eecd48c833a"). InnerVolumeSpecName "kube-api-access-6q9mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:50.330937 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.330912    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-otlp-tls-certs" pod "771b14c6-d117-4db0-85d5-7eecd48c833a" (UID: "771b14c6-d117-4db0-85d5-7eecd48c833a"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:50.331036 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.330942    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-monitoring-tls-certs" pod "771b14c6-d117-4db0-85d5-7eecd48c833a" (UID: "771b14c6-d117-4db0-85d5-7eecd48c833a"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:50.429448 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.429413    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:50.429448 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.429443    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:50.429448 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.429454    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q9mg\" (UniqueName: \"kubernetes.io/projected/771b14c6-d117-4db0-85d5-7eecd48c833a-kube-api-access-6q9mg\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:50.429682 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.429466    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:50.429682 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.429475    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/771b14c6-d117-4db0-85d5-7eecd48c833a-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:50.670113 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.670031    4960 generic.go:356] "Generic (PLEG): container finished" podID="771b14c6-d117-4db0-85d5-7eecd48c833a" containerID="cfc981edf9841516030abe14e46e27b7faf14f48b1ac2421d3244582902658d6" exitCode=0
Oct 30 04:16:50.670113 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.670078    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:50.670316 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.670118    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"771b14c6-d117-4db0-85d5-7eecd48c833a","Type":"ContainerDied","Data":"cfc981edf9841516030abe14e46e27b7faf14f48b1ac2421d3244582902658d6"}
Oct 30 04:16:50.670316 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.670173    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"771b14c6-d117-4db0-85d5-7eecd48c833a","Type":"ContainerDied","Data":"d7cb2c66b698f21eed70a9f29a825bd0d75cf5750584a52c81f4e3c3a1620334"}
Oct 30 04:16:50.670316 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.670206    4960 scope.go:122] "RemoveContainer" containerID="cfc981edf9841516030abe14e46e27b7faf14f48b1ac2421d3244582902658d6"
Oct 30 04:16:50.680434 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.680416    4960 scope.go:122] "RemoveContainer" containerID="cfc981edf9841516030abe14e46e27b7faf14f48b1ac2421d3244582902658d6"
Oct 30 04:16:50.680729 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:16:50.680708    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc981edf9841516030abe14e46e27b7faf14f48b1ac2421d3244582902658d6\": container with ID starting with cfc981edf9841516030abe14e46e27b7faf14f48b1ac2421d3244582902658d6 not found: ID does not exist" containerID="cfc981edf9841516030abe14e46e27b7faf14f48b1ac2421d3244582902658d6"
Oct 30 04:16:50.680860 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.680734    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc981edf9841516030abe14e46e27b7faf14f48b1ac2421d3244582902658d6"} err="failed to get container status \"cfc981edf9841516030abe14e46e27b7faf14f48b1ac2421d3244582902658d6\": rpc error: code = NotFound desc = could not find container \"cfc981edf9841516030abe14e46e27b7faf14f48b1ac2421d3244582902658d6\": container with ID starting with cfc981edf9841516030abe14e46e27b7faf14f48b1ac2421d3244582902658d6 not found: ID does not exist"
Oct 30 04:16:50.727972 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.727938    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:50.729971 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.729952    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:50.764875 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.764850    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:50.765240 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.765226    4960 cpu_manager.go:414] "containerMap: removing container" podUID="771b14c6-d117-4db0-85d5-7eecd48c833a" containerName="otc-container"
Oct 30 04:16:50.765294 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.765244    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="771b14c6-d117-4db0-85d5-7eecd48c833a" containerName="otc-container"
Oct 30 04:16:50.765331 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.765317    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="771b14c6-d117-4db0-85d5-7eecd48c833a" containerName="otc-container"
Oct 30 04:16:50.769487 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.769473    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:50.788001 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.787973    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:50.788387 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.788369    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:16:50.788474 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.788447    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:16:50.788575 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.788558    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:16:50.788698 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.788679    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8rxn8\""
Oct 30 04:16:50.788761 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.788735    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:16:50.788936 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.788917    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:16:50.933633 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.933550    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:50.933777 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.933625    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1bc1f85a-284f-484e-914d-e0e3d859130c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:50.933777 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.933720    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:50.933777 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.933766    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:50.933899 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.933795    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:50.933899 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:50.933839    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p74j6\" (UniqueName: \"kubernetes.io/projected/1bc1f85a-284f-484e-914d-e0e3d859130c-kube-api-access-p74j6\") pod \"data-science-collector-collector-1\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:51.034626 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.034593    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:51.034626 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.034632    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1bc1f85a-284f-484e-914d-e0e3d859130c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:51.034906 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.034674    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:51.034906 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.034801    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:51.034906 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.034839    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:51.034906 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.034876    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p74j6\" (UniqueName: \"kubernetes.io/projected/1bc1f85a-284f-484e-914d-e0e3d859130c-kube-api-access-p74j6\") pod \"data-science-collector-collector-1\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:51.035621 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.035595    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1bc1f85a-284f-484e-914d-e0e3d859130c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:51.037596 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.037569    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:51.038038 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.038014    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:51.038139 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.038112    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:51.038223 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.038202    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:51.055627 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.055603    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p74j6\" (UniqueName: \"kubernetes.io/projected/1bc1f85a-284f-484e-914d-e0e3d859130c-kube-api-access-p74j6\") pod \"data-science-collector-collector-1\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:51.082378 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.082360    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:51.375077 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.375044    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:51.375884 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:16:51.375857    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc1f85a_284f_484e_914d_e0e3d859130c.slice/crio-a617b8cf0bf49813a60703d4a6d100939a4ec91eef02133bb8fba535f6d42e12 WatchSource:0}: Error finding container a617b8cf0bf49813a60703d4a6d100939a4ec91eef02133bb8fba535f6d42e12: Status 404 returned error can't find the container with id a617b8cf0bf49813a60703d4a6d100939a4ec91eef02133bb8fba535f6d42e12
Oct 30 04:16:51.377994 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.377960    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:16:51.378076 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.378023    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:16:51.502589 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.502555    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="771b14c6-d117-4db0-85d5-7eecd48c833a" path="/var/lib/kubelet/pods/771b14c6-d117-4db0-85d5-7eecd48c833a/volumes"
Oct 30 04:16:51.678240 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.678132    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1bc1f85a-284f-484e-914d-e0e3d859130c","Type":"ContainerStarted","Data":"70ed7c75f2622e95574f03e693a337ce910f05dbc800c2c1037c051f924445f6"}
Oct 30 04:16:51.678240 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.678206    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1bc1f85a-284f-484e-914d-e0e3d859130c","Type":"ContainerStarted","Data":"a617b8cf0bf49813a60703d4a6d100939a4ec91eef02133bb8fba535f6d42e12"}
Oct 30 04:16:51.720025 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.719978    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.719964501 podStartE2EDuration="1.719964501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:51.719322872 +0000 UTC m=+5692.368371293" watchObservedRunningTime="2026-10-30 04:16:51.719964501 +0000 UTC m=+5692.369012921"
Oct 30 04:16:51.997545 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.997468    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"]
Oct 30 04:16:51.997711 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:51.997658    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq" podUID="b38749d2-d835-4cf1-970f-faa3453dac62" containerName="korrel8r" containerID="cri-o://b4e41647ccb0054b710a55305ac9d667b2f7ca060216013aad4fb208e2c17199" gracePeriod=30
Oct 30 04:16:52.244450 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.244426    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"
Oct 30 04:16:52.349890 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.349860    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b38749d2-d835-4cf1-970f-faa3453dac62-kube-api-access-q56x2\" (UniqueName: \"kubernetes.io/projected/b38749d2-d835-4cf1-970f-faa3453dac62-kube-api-access-q56x2\") pod \"b38749d2-d835-4cf1-970f-faa3453dac62\" (UID: \"b38749d2-d835-4cf1-970f-faa3453dac62\") "
Oct 30 04:16:52.350100 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.349908    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b38749d2-d835-4cf1-970f-faa3453dac62-tls\" (UniqueName: \"kubernetes.io/secret/b38749d2-d835-4cf1-970f-faa3453dac62-tls\") pod \"b38749d2-d835-4cf1-970f-faa3453dac62\" (UID: \"b38749d2-d835-4cf1-970f-faa3453dac62\") "
Oct 30 04:16:52.350100 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.349943    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b38749d2-d835-4cf1-970f-faa3453dac62-config\" (UniqueName: \"kubernetes.io/configmap/b38749d2-d835-4cf1-970f-faa3453dac62-config\") pod \"b38749d2-d835-4cf1-970f-faa3453dac62\" (UID: \"b38749d2-d835-4cf1-970f-faa3453dac62\") "
Oct 30 04:16:52.350455 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.350425    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38749d2-d835-4cf1-970f-faa3453dac62-config" pod "b38749d2-d835-4cf1-970f-faa3453dac62" (UID: "b38749d2-d835-4cf1-970f-faa3453dac62"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:52.352154 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.352122    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b38749d2-d835-4cf1-970f-faa3453dac62-tls" pod "b38749d2-d835-4cf1-970f-faa3453dac62" (UID: "b38749d2-d835-4cf1-970f-faa3453dac62"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:52.352335 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.352316    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38749d2-d835-4cf1-970f-faa3453dac62-kube-api-access-q56x2" pod "b38749d2-d835-4cf1-970f-faa3453dac62" (UID: "b38749d2-d835-4cf1-970f-faa3453dac62"). InnerVolumeSpecName "kube-api-access-q56x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:52.375127 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.375085    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:52.451247 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.451205    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q56x2\" (UniqueName: \"kubernetes.io/projected/b38749d2-d835-4cf1-970f-faa3453dac62-kube-api-access-q56x2\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.451247 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.451237    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b38749d2-d835-4cf1-970f-faa3453dac62-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.451247 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.451253    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b38749d2-d835-4cf1-970f-faa3453dac62-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.459549 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.459523    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:16:52.459791 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.459763    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6f6becf4-245d-4106-9b55-680179f120aa" containerName="alertmanager" containerID="cri-o://a1ae8ddaa6a4927c0d7d716a1b407be75e6470aac221998de7c8f837d59fe08a" gracePeriod=120
Oct 30 04:16:52.459897 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.459814    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6f6becf4-245d-4106-9b55-680179f120aa" containerName="config-reloader" containerID="cri-o://374fec369e2a0cbaa39b14fc6003afb0dfe70369dddb5cb18d676656e53b1461" gracePeriod=120
Oct 30 04:16:52.462439 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.462416    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:16:52.462761 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.462707    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="23cc19dd-d8c0-4c2f-8243-2ca505ccb347" containerName="prometheus" containerID="cri-o://3828f949825df825e934600ff2971dd1fa6a0af2573547a9d5b2f390675c608e" gracePeriod=600
Oct 30 04:16:52.463007 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.462825    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="23cc19dd-d8c0-4c2f-8243-2ca505ccb347" containerName="thanos-sidecar" containerID="cri-o://116a654479f1e8300adaad8f5a6e1be80c88948373c27a5989d93a691ade1f7f" gracePeriod=600
Oct 30 04:16:52.463007 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.462713    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="23cc19dd-d8c0-4c2f-8243-2ca505ccb347" containerName="config-reloader" containerID="cri-o://20f1c876437b544a45acd7fd519e4bd6553c40a2db9d6db882e9b81cba2c3c62" gracePeriod=600
Oct 30 04:16:52.499077 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.499051    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:52.600155 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.600096    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:52.688605 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.688575    4960 generic.go:356] "Generic (PLEG): container finished" podID="23cc19dd-d8c0-4c2f-8243-2ca505ccb347" containerID="116a654479f1e8300adaad8f5a6e1be80c88948373c27a5989d93a691ade1f7f" exitCode=0
Oct 30 04:16:52.688605 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.688602    4960 generic.go:356] "Generic (PLEG): container finished" podID="23cc19dd-d8c0-4c2f-8243-2ca505ccb347" containerID="20f1c876437b544a45acd7fd519e4bd6553c40a2db9d6db882e9b81cba2c3c62" exitCode=0
Oct 30 04:16:52.688605 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.688610    4960 generic.go:356] "Generic (PLEG): container finished" podID="23cc19dd-d8c0-4c2f-8243-2ca505ccb347" containerID="3828f949825df825e934600ff2971dd1fa6a0af2573547a9d5b2f390675c608e" exitCode=0
Oct 30 04:16:52.688862 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.688614    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"23cc19dd-d8c0-4c2f-8243-2ca505ccb347","Type":"ContainerDied","Data":"116a654479f1e8300adaad8f5a6e1be80c88948373c27a5989d93a691ade1f7f"}
Oct 30 04:16:52.688862 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.688700    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"23cc19dd-d8c0-4c2f-8243-2ca505ccb347","Type":"ContainerDied","Data":"20f1c876437b544a45acd7fd519e4bd6553c40a2db9d6db882e9b81cba2c3c62"}
Oct 30 04:16:52.688862 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.688752    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"23cc19dd-d8c0-4c2f-8243-2ca505ccb347","Type":"ContainerDied","Data":"3828f949825df825e934600ff2971dd1fa6a0af2573547a9d5b2f390675c608e"}
Oct 30 04:16:52.690119 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.690095    4960 generic.go:356] "Generic (PLEG): container finished" podID="b38749d2-d835-4cf1-970f-faa3453dac62" containerID="b4e41647ccb0054b710a55305ac9d667b2f7ca060216013aad4fb208e2c17199" exitCode=0
Oct 30 04:16:52.690900 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.690140    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"
Oct 30 04:16:52.690900 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.690209    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq" event={"ID":"b38749d2-d835-4cf1-970f-faa3453dac62","Type":"ContainerDied","Data":"b4e41647ccb0054b710a55305ac9d667b2f7ca060216013aad4fb208e2c17199"}
Oct 30 04:16:52.690900 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.690265    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq" event={"ID":"b38749d2-d835-4cf1-970f-faa3453dac62","Type":"ContainerDied","Data":"32c51451a338482ba74d559419da8c7c60faa35a6ec62ab695cad11508d18aab"}
Oct 30 04:16:52.690900 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.690298    4960 scope.go:122] "RemoveContainer" containerID="b4e41647ccb0054b710a55305ac9d667b2f7ca060216013aad4fb208e2c17199"
Oct 30 04:16:52.702698 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.702681    4960 scope.go:122] "RemoveContainer" containerID="b4e41647ccb0054b710a55305ac9d667b2f7ca060216013aad4fb208e2c17199"
Oct 30 04:16:52.702991 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:16:52.702973    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e41647ccb0054b710a55305ac9d667b2f7ca060216013aad4fb208e2c17199\": container with ID starting with b4e41647ccb0054b710a55305ac9d667b2f7ca060216013aad4fb208e2c17199 not found: ID does not exist" containerID="b4e41647ccb0054b710a55305ac9d667b2f7ca060216013aad4fb208e2c17199"
Oct 30 04:16:52.703052 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.702999    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e41647ccb0054b710a55305ac9d667b2f7ca060216013aad4fb208e2c17199"} err="failed to get container status \"b4e41647ccb0054b710a55305ac9d667b2f7ca060216013aad4fb208e2c17199\": rpc error: code = NotFound desc = could not find container \"b4e41647ccb0054b710a55305ac9d667b2f7ca060216013aad4fb208e2c17199\": container with ID starting with b4e41647ccb0054b710a55305ac9d667b2f7ca060216013aad4fb208e2c17199 not found: ID does not exist"
Oct 30 04:16:52.743977 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.743956    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"]
Oct 30 04:16:52.747648 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.747629    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-rn2fq"]
Oct 30 04:16:52.803805 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.803785    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:52.864620 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.864526    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-config\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-config\") pod \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") "
Oct 30 04:16:52.864778 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.864643    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") "
Oct 30 04:16:52.864965 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.864940    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") "
Oct 30 04:16:52.865093 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.864992    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-tls-assets\" (UniqueName: \"kubernetes.io/projected/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-tls-assets\") pod \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") "
Oct 30 04:16:52.865093 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.865026    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config\") pod \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") "
Oct 30 04:16:52.865093 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.865082    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-secret-prometheus-web-tls-ca\") pod \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") "
Oct 30 04:16:52.865286 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.865115    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") "
Oct 30 04:16:52.865286 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.865158    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-kube-api-access-cz62l\" (UniqueName: \"kubernetes.io/projected/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-kube-api-access-cz62l\") pod \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") "
Oct 30 04:16:52.865286 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.865205    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-thanos-prometheus-http-client-file\") pod \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") "
Oct 30 04:16:52.865286 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.865240    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") "
Oct 30 04:16:52.866265 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.866238    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") "
Oct 30 04:16:52.866378 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.866292    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-config-out\" (UniqueName: \"kubernetes.io/empty-dir/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-config-out\") pod \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") "
Oct 30 04:16:52.866378 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.866328    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\" (UID: \"23cc19dd-d8c0-4c2f-8243-2ca505ccb347\") "
Oct 30 04:16:52.866989 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.866895    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-2" pod "23cc19dd-d8c0-4c2f-8243-2ca505ccb347" (UID: "23cc19dd-d8c0-4c2f-8243-2ca505ccb347"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:52.867190 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.867128    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-0" pod "23cc19dd-d8c0-4c2f-8243-2ca505ccb347" (UID: "23cc19dd-d8c0-4c2f-8243-2ca505ccb347"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:52.867692 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.867661    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-1" pod "23cc19dd-d8c0-4c2f-8243-2ca505ccb347" (UID: "23cc19dd-d8c0-4c2f-8243-2ca505ccb347"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:52.869068 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.868563    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-config" pod "23cc19dd-d8c0-4c2f-8243-2ca505ccb347" (UID: "23cc19dd-d8c0-4c2f-8243-2ca505ccb347"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:52.869068 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.868595    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.869068 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.868620    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.869068 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.868637    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.871954 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.871920    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-secret-prometheus-web-tls-ca" pod "23cc19dd-d8c0-4c2f-8243-2ca505ccb347" (UID: "23cc19dd-d8c0-4c2f-8243-2ca505ccb347"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:52.872042 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.872017    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-config-out" pod "23cc19dd-d8c0-4c2f-8243-2ca505ccb347" (UID: "23cc19dd-d8c0-4c2f-8243-2ca505ccb347"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:52.872199 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.872133    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "23cc19dd-d8c0-4c2f-8243-2ca505ccb347" (UID: "23cc19dd-d8c0-4c2f-8243-2ca505ccb347"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:52.872326 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.872310    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-tls-assets" pod "23cc19dd-d8c0-4c2f-8243-2ca505ccb347" (UID: "23cc19dd-d8c0-4c2f-8243-2ca505ccb347"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:52.872697 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.872674    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "23cc19dd-d8c0-4c2f-8243-2ca505ccb347" (UID: "23cc19dd-d8c0-4c2f-8243-2ca505ccb347"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:52.872697 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.872678    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-kube-api-access-cz62l" pod "23cc19dd-d8c0-4c2f-8243-2ca505ccb347" (UID: "23cc19dd-d8c0-4c2f-8243-2ca505ccb347"). InnerVolumeSpecName "kube-api-access-cz62l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:52.872870 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.872706    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-thanos-prometheus-http-client-file" pod "23cc19dd-d8c0-4c2f-8243-2ca505ccb347" (UID: "23cc19dd-d8c0-4c2f-8243-2ca505ccb347"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:52.874979 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.874955    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87" pod "23cc19dd-d8c0-4c2f-8243-2ca505ccb347" (UID: "23cc19dd-d8c0-4c2f-8243-2ca505ccb347"). InnerVolumeSpecName "pvc-9689c530-7f91-4584-b28e-8809bed9e799". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:16:52.884412 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.884392    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config" pod "23cc19dd-d8c0-4c2f-8243-2ca505ccb347" (UID: "23cc19dd-d8c0-4c2f-8243-2ca505ccb347"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:52.969337 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.969308    4960 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-secret-prometheus-web-tls-ca\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.969337 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.969334    4960 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.969489 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.969345    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz62l\" (UniqueName: \"kubernetes.io/projected/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-kube-api-access-cz62l\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.969489 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.969355    4960 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.969489 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.969365    4960 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-config-out\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.969489 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.969373    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.969489 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.969382    4960 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.969489 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.969408    4960 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") on node \"ip-10-0-128-241.ec2.internal\" "
Oct 30 04:16:52.969489 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.969416    4960 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-tls-assets\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.969489 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.969424    4960 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23cc19dd-d8c0-4c2f-8243-2ca505ccb347-web-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.980958 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:52.980939    4960 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-9689c530-7f91-4584-b28e-8809bed9e799" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87") on node "ip-10-0-128-241.ec2.internal"
Oct 30 04:16:53.074678 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.074645    4960 reconciler_common.go:299] "Volume detached for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"csi-6f8c8e4c1326657251c30594036033694cadf25e97408e4f9d9da436f78f3fa4\""
Oct 30 04:16:53.098069 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.098035    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:53.119840 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.119807    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:53.226021 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.225991    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:53.339330 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.339294    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:53.504401 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.504329    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b38749d2-d835-4cf1-970f-faa3453dac62" path="/var/lib/kubelet/pods/b38749d2-d835-4cf1-970f-faa3453dac62/volumes"
Oct 30 04:16:53.697717 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.697681    4960 generic.go:356] "Generic (PLEG): container finished" podID="6f6becf4-245d-4106-9b55-680179f120aa" containerID="374fec369e2a0cbaa39b14fc6003afb0dfe70369dddb5cb18d676656e53b1461" exitCode=0
Oct 30 04:16:53.697889 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.697762    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6f6becf4-245d-4106-9b55-680179f120aa","Type":"ContainerDied","Data":"374fec369e2a0cbaa39b14fc6003afb0dfe70369dddb5cb18d676656e53b1461"}
Oct 30 04:16:53.700440 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.700403    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"23cc19dd-d8c0-4c2f-8243-2ca505ccb347","Type":"ContainerDied","Data":"d4ea1135e54580bd7440dd5e8cd880d6fd34d74f1082a61e23ef86571cf4b4b1"}
Oct 30 04:16:53.700568 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.700414    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:53.700568 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.700482    4960 scope.go:122] "RemoveContainer" containerID="116a654479f1e8300adaad8f5a6e1be80c88948373c27a5989d93a691ade1f7f"
Oct 30 04:16:53.701498 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.701452    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="1bc1f85a-284f-484e-914d-e0e3d859130c" containerName="otc-container" containerID="cri-o://70ed7c75f2622e95574f03e693a337ce910f05dbc800c2c1037c051f924445f6" gracePeriod=30
Oct 30 04:16:53.709889 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.709871    4960 scope.go:122] "RemoveContainer" containerID="20f1c876437b544a45acd7fd519e4bd6553c40a2db9d6db882e9b81cba2c3c62"
Oct 30 04:16:53.767454 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.767434    4960 scope.go:122] "RemoveContainer" containerID="3828f949825df825e934600ff2971dd1fa6a0af2573547a9d5b2f390675c608e"
Oct 30 04:16:53.798567 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.798543    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:16:53.812326 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.812304    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:16:53.813386 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.813369    4960 scope.go:122] "RemoveContainer" containerID="742d5016609a7cfe13dc8e8e5528f5f83268a68acb4bf3ff4ed1dbfce8e07c43"
Oct 30 04:16:53.956468 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.956445    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:53.956606 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:16:53.956518    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:16:53.983323 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.983295    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc1f85a-284f-484e-914d-e0e3d859130c-kube-api-access-p74j6\" (UniqueName: \"kubernetes.io/projected/1bc1f85a-284f-484e-914d-e0e3d859130c-kube-api-access-p74j6\") pod \"1bc1f85a-284f-484e-914d-e0e3d859130c\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") "
Oct 30 04:16:53.983488 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.983348    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-data-science-collector-ta-client-cert\") pod \"1bc1f85a-284f-484e-914d-e0e3d859130c\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") "
Oct 30 04:16:53.983488 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.983384    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-monitoring-tls-certs\") pod \"1bc1f85a-284f-484e-914d-e0e3d859130c\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") "
Oct 30 04:16:53.983488 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.983417    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-tls-certs\") pod \"1bc1f85a-284f-484e-914d-e0e3d859130c\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") "
Oct 30 04:16:53.983666 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.983525    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bc1f85a-284f-484e-914d-e0e3d859130c-otc-internal\" (UniqueName: \"kubernetes.io/configmap/1bc1f85a-284f-484e-914d-e0e3d859130c-otc-internal\") pod \"1bc1f85a-284f-484e-914d-e0e3d859130c\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") "
Oct 30 04:16:53.983666 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.983573    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-otlp-tls-certs\") pod \"1bc1f85a-284f-484e-914d-e0e3d859130c\" (UID: \"1bc1f85a-284f-484e-914d-e0e3d859130c\") "
Oct 30 04:16:53.984495 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.984456    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bc1f85a-284f-484e-914d-e0e3d859130c-otc-internal" pod "1bc1f85a-284f-484e-914d-e0e3d859130c" (UID: "1bc1f85a-284f-484e-914d-e0e3d859130c"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:53.986281 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.986254    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-monitoring-tls-certs" pod "1bc1f85a-284f-484e-914d-e0e3d859130c" (UID: "1bc1f85a-284f-484e-914d-e0e3d859130c"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:53.986633 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.986595    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-tls-certs" pod "1bc1f85a-284f-484e-914d-e0e3d859130c" (UID: "1bc1f85a-284f-484e-914d-e0e3d859130c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:53.986889 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.986862    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-data-science-collector-ta-client-cert" pod "1bc1f85a-284f-484e-914d-e0e3d859130c" (UID: "1bc1f85a-284f-484e-914d-e0e3d859130c"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:53.987056 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.987033    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc1f85a-284f-484e-914d-e0e3d859130c-kube-api-access-p74j6" pod "1bc1f85a-284f-484e-914d-e0e3d859130c" (UID: "1bc1f85a-284f-484e-914d-e0e3d859130c"). InnerVolumeSpecName "kube-api-access-p74j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:53.987336 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:53.987310    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-otlp-tls-certs" pod "1bc1f85a-284f-484e-914d-e0e3d859130c" (UID: "1bc1f85a-284f-484e-914d-e0e3d859130c"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:54.037897 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:54.037780    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:54.084166 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:54.084135    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p74j6\" (UniqueName: \"kubernetes.io/projected/1bc1f85a-284f-484e-914d-e0e3d859130c-kube-api-access-p74j6\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:54.084166 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:54.084161    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:54.084166 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:54.084172    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:54.084166 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:54.084199    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:54.084536 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:54.084210    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1bc1f85a-284f-484e-914d-e0e3d859130c-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:54.084536 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:54.084218    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc1f85a-284f-484e-914d-e0e3d859130c-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:16:54.155171 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:54.155137    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:54.261142 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:54.261107    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:54.710227 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:54.710174    4960 generic.go:356] "Generic (PLEG): container finished" podID="1bc1f85a-284f-484e-914d-e0e3d859130c" containerID="70ed7c75f2622e95574f03e693a337ce910f05dbc800c2c1037c051f924445f6" exitCode=0
Oct 30 04:16:54.710227 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:54.710227    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1bc1f85a-284f-484e-914d-e0e3d859130c","Type":"ContainerDied","Data":"70ed7c75f2622e95574f03e693a337ce910f05dbc800c2c1037c051f924445f6"}
Oct 30 04:16:54.710756 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:54.710233    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:16:54.710756 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:54.710267    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1bc1f85a-284f-484e-914d-e0e3d859130c","Type":"ContainerDied","Data":"a617b8cf0bf49813a60703d4a6d100939a4ec91eef02133bb8fba535f6d42e12"}
Oct 30 04:16:54.710756 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:54.710290    4960 scope.go:122] "RemoveContainer" containerID="70ed7c75f2622e95574f03e693a337ce910f05dbc800c2c1037c051f924445f6"
Oct 30 04:16:54.720149 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:54.720129    4960 scope.go:122] "RemoveContainer" containerID="70ed7c75f2622e95574f03e693a337ce910f05dbc800c2c1037c051f924445f6"
Oct 30 04:16:54.720451 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:16:54.720430    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ed7c75f2622e95574f03e693a337ce910f05dbc800c2c1037c051f924445f6\": container with ID starting with 70ed7c75f2622e95574f03e693a337ce910f05dbc800c2c1037c051f924445f6 not found: ID does not exist" containerID="70ed7c75f2622e95574f03e693a337ce910f05dbc800c2c1037c051f924445f6"
Oct 30 04:16:54.720502 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:54.720460    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ed7c75f2622e95574f03e693a337ce910f05dbc800c2c1037c051f924445f6"} err="failed to get container status \"70ed7c75f2622e95574f03e693a337ce910f05dbc800c2c1037c051f924445f6\": rpc error: code = NotFound desc = could not find container \"70ed7c75f2622e95574f03e693a337ce910f05dbc800c2c1037c051f924445f6\": container with ID starting with 70ed7c75f2622e95574f03e693a337ce910f05dbc800c2c1037c051f924445f6 not found: ID does not exist"
Oct 30 04:16:54.763625 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:54.763589    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:54.766679 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:54.766657    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:16:55.502277 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:55.502247    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc1f85a-284f-484e-914d-e0e3d859130c" path="/var/lib/kubelet/pods/1bc1f85a-284f-484e-914d-e0e3d859130c/volumes"
Oct 30 04:16:55.502665 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:55.502653    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23cc19dd-d8c0-4c2f-8243-2ca505ccb347" path="/var/lib/kubelet/pods/23cc19dd-d8c0-4c2f-8243-2ca505ccb347/volumes"
Oct 30 04:16:56.519595 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:16:56.519557    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6f6becf4-245d-4106-9b55-680179f120aa" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.134:9093/-/ready\": dial tcp 10.132.0.134:9093: connect: connection refused"
Oct 30 04:17:00.071893 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:00.071869    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 04:17:00.078045 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:00.078023    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 04:17:00.080742 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:00.080726    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 04:17:00.086561 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:00.086545    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fl65k_3c022b3d-db3f-45a7-b6b7-5404c6f170c8/ovn-acl-logging/0.log"
Oct 30 04:17:00.396153 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:00.396078    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:00.670172 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:00.670089    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:00.798783 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:00.798755    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:00.936239 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:00.936142    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:01.074712 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.074680    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:01.118467 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.118439    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"]
Oct 30 04:17:01.118805 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.118792    4960 cpu_manager.go:414] "containerMap: removing container" podUID="23cc19dd-d8c0-4c2f-8243-2ca505ccb347" containerName="init-config-reloader"
Oct 30 04:17:01.118860 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.118807    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23cc19dd-d8c0-4c2f-8243-2ca505ccb347" containerName="init-config-reloader"
Oct 30 04:17:01.118860 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.118824    4960 cpu_manager.go:414] "containerMap: removing container" podUID="23cc19dd-d8c0-4c2f-8243-2ca505ccb347" containerName="config-reloader"
Oct 30 04:17:01.118860 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.118829    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23cc19dd-d8c0-4c2f-8243-2ca505ccb347" containerName="config-reloader"
Oct 30 04:17:01.118860 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.118835    4960 cpu_manager.go:414] "containerMap: removing container" podUID="1bc1f85a-284f-484e-914d-e0e3d859130c" containerName="otc-container"
Oct 30 04:17:01.118860 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.118840    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc1f85a-284f-484e-914d-e0e3d859130c" containerName="otc-container"
Oct 30 04:17:01.118860 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.118849    4960 cpu_manager.go:414] "containerMap: removing container" podUID="23cc19dd-d8c0-4c2f-8243-2ca505ccb347" containerName="prometheus"
Oct 30 04:17:01.118860 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.118854    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23cc19dd-d8c0-4c2f-8243-2ca505ccb347" containerName="prometheus"
Oct 30 04:17:01.119076 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.118869    4960 cpu_manager.go:414] "containerMap: removing container" podUID="b38749d2-d835-4cf1-970f-faa3453dac62" containerName="korrel8r"
Oct 30 04:17:01.119076 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.118873    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b38749d2-d835-4cf1-970f-faa3453dac62" containerName="korrel8r"
Oct 30 04:17:01.119076 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.118881    4960 cpu_manager.go:414] "containerMap: removing container" podUID="23cc19dd-d8c0-4c2f-8243-2ca505ccb347" containerName="thanos-sidecar"
Oct 30 04:17:01.119076 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.118885    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23cc19dd-d8c0-4c2f-8243-2ca505ccb347" containerName="thanos-sidecar"
Oct 30 04:17:01.119076 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.118932    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="23cc19dd-d8c0-4c2f-8243-2ca505ccb347" containerName="prometheus"
Oct 30 04:17:01.119076 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.118942    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="b38749d2-d835-4cf1-970f-faa3453dac62" containerName="korrel8r"
Oct 30 04:17:01.119076 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.118948    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="23cc19dd-d8c0-4c2f-8243-2ca505ccb347" containerName="thanos-sidecar"
Oct 30 04:17:01.119076 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.118957    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="23cc19dd-d8c0-4c2f-8243-2ca505ccb347" containerName="config-reloader"
Oct 30 04:17:01.119076 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.118963    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc1f85a-284f-484e-914d-e0e3d859130c" containerName="otc-container"
Oct 30 04:17:01.123616 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.123599    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"
Oct 30 04:17:01.142109 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.142083    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"]
Oct 30 04:17:01.205264 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.205202    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\""
Oct 30 04:17:01.205264 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.205217    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-hxkgl\""
Oct 30 04:17:01.205264 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.205203    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\""
Oct 30 04:17:01.241683 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.241659    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-config\") pod \"korrel8r-5cbcc89b8f-gl24q\" (UID: \"8281b1f0-73c3-47c8-b82e-aabc93dfe3dd\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"
Oct 30 04:17:01.241777 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.241706    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bjjj\" (UniqueName: \"kubernetes.io/projected/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-kube-api-access-7bjjj\") pod \"korrel8r-5cbcc89b8f-gl24q\" (UID: \"8281b1f0-73c3-47c8-b82e-aabc93dfe3dd\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"
Oct 30 04:17:01.241829 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.241775    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-tls\") pod \"korrel8r-5cbcc89b8f-gl24q\" (UID: \"8281b1f0-73c3-47c8-b82e-aabc93dfe3dd\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"
Oct 30 04:17:01.343075 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.343044    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bjjj\" (UniqueName: \"kubernetes.io/projected/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-kube-api-access-7bjjj\") pod \"korrel8r-5cbcc89b8f-gl24q\" (UID: \"8281b1f0-73c3-47c8-b82e-aabc93dfe3dd\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"
Oct 30 04:17:01.343207 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.343087    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-tls\") pod \"korrel8r-5cbcc89b8f-gl24q\" (UID: \"8281b1f0-73c3-47c8-b82e-aabc93dfe3dd\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"
Oct 30 04:17:01.343207 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.343122    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-config\") pod \"korrel8r-5cbcc89b8f-gl24q\" (UID: \"8281b1f0-73c3-47c8-b82e-aabc93dfe3dd\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"
Oct 30 04:17:01.343841 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.343815    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-config\") pod \"korrel8r-5cbcc89b8f-gl24q\" (UID: \"8281b1f0-73c3-47c8-b82e-aabc93dfe3dd\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"
Oct 30 04:17:01.345756 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.345731    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-tls\") pod \"korrel8r-5cbcc89b8f-gl24q\" (UID: \"8281b1f0-73c3-47c8-b82e-aabc93dfe3dd\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"
Oct 30 04:17:01.351617 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.351592    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:01.370070 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.370044    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bjjj\" (UniqueName: \"kubernetes.io/projected/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-kube-api-access-7bjjj\") pod \"korrel8r-5cbcc89b8f-gl24q\" (UID: \"8281b1f0-73c3-47c8-b82e-aabc93dfe3dd\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"
Oct 30 04:17:01.436522 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.436502    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"
Oct 30 04:17:01.520037 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.520004    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6f6becf4-245d-4106-9b55-680179f120aa" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.134:9093/-/ready\": dial tcp 10.132.0.134:9093: connect: connection refused"
Oct 30 04:17:01.685637 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.685597    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:01.728028 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.728008    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"]
Oct 30 04:17:01.746804 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.746779    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q" event={"ID":"8281b1f0-73c3-47c8-b82e-aabc93dfe3dd","Type":"ContainerStarted","Data":"c6d9c755084599ceac10d3d1a3f1d83afb6966a21b49b05e7577d0b3a2053010"}
Oct 30 04:17:01.785867 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.785824    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:01.914475 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:01.914444    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:02.624126 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.624098    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:17:02.627580 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.627557    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.635815 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.635511    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:02.655284 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.655251    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f943646f-688d-41b9-9b16-4d54b4195bc2-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.655446 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.655357    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.655446 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.655411    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7wqt\" (UniqueName: \"kubernetes.io/projected/f943646f-688d-41b9-9b16-4d54b4195bc2-kube-api-access-f7wqt\") pod \"data-science-collector-collector-1\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.655582 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.655466    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.655582 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.655542    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.655582 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.655574    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.659971 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.659945    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:17:02.660125 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.659982    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:17:02.661064 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.661039    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:17:02.662278 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.662258    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-w4g8j\""
Oct 30 04:17:02.662392 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.662307    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:17:02.718793 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.718761    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:17:02.718793 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.718769    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:17:02.743016 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.742989    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:02.753853 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.753824    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q" event={"ID":"8281b1f0-73c3-47c8-b82e-aabc93dfe3dd","Type":"ContainerStarted","Data":"4451d6c8d9d0831f538aa93b989a1f6b6b399fd5178a55b7978cc08257cfe35e"}
Oct 30 04:17:02.754000 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.753890    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"
Oct 30 04:17:02.756744 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.755997    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.756744 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.756055    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.756744 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.756093    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.756744 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.756124    4960 generic.go:356] "Generic (PLEG): container finished" podID="6f6becf4-245d-4106-9b55-680179f120aa" containerID="a1ae8ddaa6a4927c0d7d716a1b407be75e6470aac221998de7c8f837d59fe08a" exitCode=0
Oct 30 04:17:02.756744 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.756136    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f943646f-688d-41b9-9b16-4d54b4195bc2-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.756744 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.756241    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.756744 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.756284    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7wqt\" (UniqueName: \"kubernetes.io/projected/f943646f-688d-41b9-9b16-4d54b4195bc2-kube-api-access-f7wqt\") pod \"data-science-collector-collector-1\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.756744 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.756305    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:02.756744 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.756361    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6f6becf4-245d-4106-9b55-680179f120aa","Type":"ContainerDied","Data":"a1ae8ddaa6a4927c0d7d716a1b407be75e6470aac221998de7c8f837d59fe08a"}
Oct 30 04:17:02.756744 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.756427    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6f6becf4-245d-4106-9b55-680179f120aa","Type":"ContainerDied","Data":"b46a0f2c90698d2a19975d45960b4d12089e29073130ca30ff5399e6e2b2426e"}
Oct 30 04:17:02.756744 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.756467    4960 scope.go:122] "RemoveContainer" containerID="374fec369e2a0cbaa39b14fc6003afb0dfe70369dddb5cb18d676656e53b1461"
Oct 30 04:17:02.757939 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.757792    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f943646f-688d-41b9-9b16-4d54b4195bc2-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.759117 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.759084    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.759323 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.759305    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.759457 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.759435    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.760906 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.760886    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.764535 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.764510    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:02.773147 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.773121    4960 scope.go:122] "RemoveContainer" containerID="a1ae8ddaa6a4927c0d7d716a1b407be75e6470aac221998de7c8f837d59fe08a"
Oct 30 04:17:02.777877 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.777855    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7wqt\" (UniqueName: \"kubernetes.io/projected/f943646f-688d-41b9-9b16-4d54b4195bc2-kube-api-access-f7wqt\") pod \"data-science-collector-collector-1\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.781024 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.781009    4960 scope.go:122] "RemoveContainer" containerID="c13f3958fe3983d4856457b38683bc01d1184e7fb282a0f8a6973fc65c6dcfdf"
Oct 30 04:17:02.788743 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.788709    4960 scope.go:122] "RemoveContainer" containerID="374fec369e2a0cbaa39b14fc6003afb0dfe70369dddb5cb18d676656e53b1461"
Oct 30 04:17:02.789011 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:17:02.788991    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"374fec369e2a0cbaa39b14fc6003afb0dfe70369dddb5cb18d676656e53b1461\": container with ID starting with 374fec369e2a0cbaa39b14fc6003afb0dfe70369dddb5cb18d676656e53b1461 not found: ID does not exist" containerID="374fec369e2a0cbaa39b14fc6003afb0dfe70369dddb5cb18d676656e53b1461"
Oct 30 04:17:02.789066 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.789018    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374fec369e2a0cbaa39b14fc6003afb0dfe70369dddb5cb18d676656e53b1461"} err="failed to get container status \"374fec369e2a0cbaa39b14fc6003afb0dfe70369dddb5cb18d676656e53b1461\": rpc error: code = NotFound desc = could not find container \"374fec369e2a0cbaa39b14fc6003afb0dfe70369dddb5cb18d676656e53b1461\": container with ID starting with 374fec369e2a0cbaa39b14fc6003afb0dfe70369dddb5cb18d676656e53b1461 not found: ID does not exist"
Oct 30 04:17:02.789066 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.789035    4960 scope.go:122] "RemoveContainer" containerID="a1ae8ddaa6a4927c0d7d716a1b407be75e6470aac221998de7c8f837d59fe08a"
Oct 30 04:17:02.789372 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:17:02.789356    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1ae8ddaa6a4927c0d7d716a1b407be75e6470aac221998de7c8f837d59fe08a\": container with ID starting with a1ae8ddaa6a4927c0d7d716a1b407be75e6470aac221998de7c8f837d59fe08a not found: ID does not exist" containerID="a1ae8ddaa6a4927c0d7d716a1b407be75e6470aac221998de7c8f837d59fe08a"
Oct 30 04:17:02.789423 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.789378    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ae8ddaa6a4927c0d7d716a1b407be75e6470aac221998de7c8f837d59fe08a"} err="failed to get container status \"a1ae8ddaa6a4927c0d7d716a1b407be75e6470aac221998de7c8f837d59fe08a\": rpc error: code = NotFound desc = could not find container \"a1ae8ddaa6a4927c0d7d716a1b407be75e6470aac221998de7c8f837d59fe08a\": container with ID starting with a1ae8ddaa6a4927c0d7d716a1b407be75e6470aac221998de7c8f837d59fe08a not found: ID does not exist"
Oct 30 04:17:02.789423 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.789400    4960 scope.go:122] "RemoveContainer" containerID="c13f3958fe3983d4856457b38683bc01d1184e7fb282a0f8a6973fc65c6dcfdf"
Oct 30 04:17:02.789663 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:17:02.789645    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c13f3958fe3983d4856457b38683bc01d1184e7fb282a0f8a6973fc65c6dcfdf\": container with ID starting with c13f3958fe3983d4856457b38683bc01d1184e7fb282a0f8a6973fc65c6dcfdf not found: ID does not exist" containerID="c13f3958fe3983d4856457b38683bc01d1184e7fb282a0f8a6973fc65c6dcfdf"
Oct 30 04:17:02.789704 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.789668    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c13f3958fe3983d4856457b38683bc01d1184e7fb282a0f8a6973fc65c6dcfdf"} err="failed to get container status \"c13f3958fe3983d4856457b38683bc01d1184e7fb282a0f8a6973fc65c6dcfdf\": rpc error: code = NotFound desc = could not find container \"c13f3958fe3983d4856457b38683bc01d1184e7fb282a0f8a6973fc65c6dcfdf\": container with ID starting with c13f3958fe3983d4856457b38683bc01d1184e7fb282a0f8a6973fc65c6dcfdf not found: ID does not exist"
Oct 30 04:17:02.852930 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.852885    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q" podStartSLOduration=1.852873893 podStartE2EDuration="1.852873893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:02.851935958 +0000 UTC m=+5703.500984401" watchObservedRunningTime="2026-10-30 04:17:02.852873893 +0000 UTC m=+5703.501922308"
Oct 30 04:17:02.857139 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.857114    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-cluster-tls-config\") pod \"6f6becf4-245d-4106-9b55-680179f120aa\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") "
Oct 30 04:17:02.857290 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.857155    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f6becf4-245d-4106-9b55-680179f120aa-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f6becf4-245d-4106-9b55-680179f120aa-config-out\") pod \"6f6becf4-245d-4106-9b55-680179f120aa\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") "
Oct 30 04:17:02.857290 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.857254    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f6becf4-245d-4106-9b55-680179f120aa-tls-assets\" (UniqueName: \"kubernetes.io/projected/6f6becf4-245d-4106-9b55-680179f120aa-tls-assets\") pod \"6f6becf4-245d-4106-9b55-680179f120aa\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") "
Oct 30 04:17:02.857389 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.857296    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f6becf4-245d-4106-9b55-680179f120aa-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6f6becf4-245d-4106-9b55-680179f120aa-alertmanager-data-science-monitoringstack-db\") pod \"6f6becf4-245d-4106-9b55-680179f120aa\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") "
Oct 30 04:17:02.857389 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.857342    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-config-volume\" (UniqueName: \"kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-config-volume\") pod \"6f6becf4-245d-4106-9b55-680179f120aa\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") "
Oct 30 04:17:02.857505 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.857409    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-web-config\" (UniqueName: \"kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-web-config\") pod \"6f6becf4-245d-4106-9b55-680179f120aa\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") "
Oct 30 04:17:02.857505 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.857437    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f6becf4-245d-4106-9b55-680179f120aa-kube-api-access-bltj5\" (UniqueName: \"kubernetes.io/projected/6f6becf4-245d-4106-9b55-680179f120aa-kube-api-access-bltj5\") pod \"6f6becf4-245d-4106-9b55-680179f120aa\" (UID: \"6f6becf4-245d-4106-9b55-680179f120aa\") "
Oct 30 04:17:02.857915 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.857884    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6becf4-245d-4106-9b55-680179f120aa-alertmanager-data-science-monitoringstack-db" pod "6f6becf4-245d-4106-9b55-680179f120aa" (UID: "6f6becf4-245d-4106-9b55-680179f120aa"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:02.859567 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.859539    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6becf4-245d-4106-9b55-680179f120aa-config-out" pod "6f6becf4-245d-4106-9b55-680179f120aa" (UID: "6f6becf4-245d-4106-9b55-680179f120aa"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:02.859926 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.859909    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f6becf4-245d-4106-9b55-680179f120aa-tls-assets" pod "6f6becf4-245d-4106-9b55-680179f120aa" (UID: "6f6becf4-245d-4106-9b55-680179f120aa"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:02.860078 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.860055    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-config-volume" pod "6f6becf4-245d-4106-9b55-680179f120aa" (UID: "6f6becf4-245d-4106-9b55-680179f120aa"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:02.860438 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.860423    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f6becf4-245d-4106-9b55-680179f120aa-kube-api-access-bltj5" pod "6f6becf4-245d-4106-9b55-680179f120aa" (UID: "6f6becf4-245d-4106-9b55-680179f120aa"). InnerVolumeSpecName "kube-api-access-bltj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:02.864115 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.864090    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-cluster-tls-config" pod "6f6becf4-245d-4106-9b55-680179f120aa" (UID: "6f6becf4-245d-4106-9b55-680179f120aa"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:02.871712 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.871692    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-web-config" pod "6f6becf4-245d-4106-9b55-680179f120aa" (UID: "6f6becf4-245d-4106-9b55-680179f120aa"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:02.878711 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.878691    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:02.943014 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.942996    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:02.958558 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.958539    4960 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-config-volume\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:02.958558 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.958559    4960 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-web-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:02.958677 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.958568    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bltj5\" (UniqueName: \"kubernetes.io/projected/6f6becf4-245d-4106-9b55-680179f120aa-kube-api-access-bltj5\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:02.958677 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.958577    4960 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6f6becf4-245d-4106-9b55-680179f120aa-cluster-tls-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:02.958677 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.958586    4960 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f6becf4-245d-4106-9b55-680179f120aa-config-out\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:02.958677 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.958596    4960 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f6becf4-245d-4106-9b55-680179f120aa-tls-assets\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:02.958677 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:02.958604    4960 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6f6becf4-245d-4106-9b55-680179f120aa-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:03.108464 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:03.108424    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:17:03.111220 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:03.111190    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:17:03.236171 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:03.236149    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:17:03.237672 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:17:03.237648    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf943646f_688d_41b9_9b16_4d54b4195bc2.slice/crio-40b16023d827a87c67490928f77a34100e0a6d31bc01c826d14369127e36ddf5 WatchSource:0}: Error finding container 40b16023d827a87c67490928f77a34100e0a6d31bc01c826d14369127e36ddf5: Status 404 returned error can't find the container with id 40b16023d827a87c67490928f77a34100e0a6d31bc01c826d14369127e36ddf5
Oct 30 04:17:03.239973 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:03.239933    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:17:03.240070 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:03.239994    4960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 04:17:03.503017 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:03.502946    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f6becf4-245d-4106-9b55-680179f120aa" path="/var/lib/kubelet/pods/6f6becf4-245d-4106-9b55-680179f120aa/volumes"
Oct 30 04:17:03.763456 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:03.763375    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f943646f-688d-41b9-9b16-4d54b4195bc2","Type":"ContainerStarted","Data":"c887ea279eb15539b3d79ac80cccc0da79c341bf2f12211dfef6cb8d4bf87568"}
Oct 30 04:17:03.763456 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:03.763428    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f943646f-688d-41b9-9b16-4d54b4195bc2","Type":"ContainerStarted","Data":"40b16023d827a87c67490928f77a34100e0a6d31bc01c826d14369127e36ddf5"}
Oct 30 04:17:03.808725 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:03.808680    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.808666828 podStartE2EDuration="1.808666828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:03.807523099 +0000 UTC m=+5704.456571520" watchObservedRunningTime="2026-10-30 04:17:03.808666828 +0000 UTC m=+5704.457715300"
Oct 30 04:17:05.509262 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:05.509218    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:05.751894 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:05.751862    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:05.922833 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:05.922706    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:06.203173 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.203091    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:17:06.203620 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.203597    4960 cpu_manager.go:414] "containerMap: removing container" podUID="6f6becf4-245d-4106-9b55-680179f120aa" containerName="init-config-reloader"
Oct 30 04:17:06.203620 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.203621    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f6becf4-245d-4106-9b55-680179f120aa" containerName="init-config-reloader"
Oct 30 04:17:06.203799 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.203636    4960 cpu_manager.go:414] "containerMap: removing container" podUID="6f6becf4-245d-4106-9b55-680179f120aa" containerName="alertmanager"
Oct 30 04:17:06.203799 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.203643    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f6becf4-245d-4106-9b55-680179f120aa" containerName="alertmanager"
Oct 30 04:17:06.203799 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.203657    4960 cpu_manager.go:414] "containerMap: removing container" podUID="6f6becf4-245d-4106-9b55-680179f120aa" containerName="config-reloader"
Oct 30 04:17:06.203799 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.203665    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f6becf4-245d-4106-9b55-680179f120aa" containerName="config-reloader"
Oct 30 04:17:06.203799 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.203769    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f6becf4-245d-4106-9b55-680179f120aa" containerName="alertmanager"
Oct 30 04:17:06.204069 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.203816    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f6becf4-245d-4106-9b55-680179f120aa" containerName="config-reloader"
Oct 30 04:17:06.212684 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.212660    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.234820 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.234797    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-rfjld\""
Oct 30 04:17:06.234820 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.234799    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:17:06.238783 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.238762    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\""
Oct 30 04:17:06.240685 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.240665    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:06.247772 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.246297    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\""
Oct 30 04:17:06.247772 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.246359    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\""
Oct 30 04:17:06.247961 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.247843    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:17:06.288281 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.288249    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/400a4c20-1943-4e25-8816-2240d796b09b-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.288417 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.288288    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hgcw\" (UniqueName: \"kubernetes.io/projected/400a4c20-1943-4e25-8816-2240d796b09b-kube-api-access-7hgcw\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.288466 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.288429    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.288466 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.288455    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.288537 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.288519    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/400a4c20-1943-4e25-8816-2240d796b09b-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.288591 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.288579    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/400a4c20-1943-4e25-8816-2240d796b09b-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.288652 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.288636    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.356523 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.356485    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:17:06.360320 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.360301    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.380891 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.380864    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 04:17:06.380994 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.380922    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 04:17:06.381060 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.381044    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 04:17:06.389350 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.389330    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.389409 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.389365    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/400a4c20-1943-4e25-8816-2240d796b09b-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.389409 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.389386    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hgcw\" (UniqueName: \"kubernetes.io/projected/400a4c20-1943-4e25-8816-2240d796b09b-kube-api-access-7hgcw\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.389478 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.389426    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.389478 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.389442    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.389478 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.389462    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/400a4c20-1943-4e25-8816-2240d796b09b-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.389618 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.389524    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/400a4c20-1943-4e25-8816-2240d796b09b-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.389786 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.389764    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/400a4c20-1943-4e25-8816-2240d796b09b-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.392134 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.392109    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/400a4c20-1943-4e25-8816-2240d796b09b-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.392267 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.392233    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/400a4c20-1943-4e25-8816-2240d796b09b-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.392342 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.392266    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.392650 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.392631    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.392693 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.392664    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.401808 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.401784    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:17:06.419515 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.419495    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hgcw\" (UniqueName: \"kubernetes.io/projected/400a4c20-1943-4e25-8816-2240d796b09b-kube-api-access-7hgcw\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.443154 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.443100    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 04:17:06.443282 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.443159    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 04:17:06.443349 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.443281    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:17:06.446157 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.446126    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:06.454546 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.454487    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 04:17:06.454635 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.454569    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:17:06.454677 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.454653    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-dbprj\""
Oct 30 04:17:06.454733 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.454714    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 04:17:06.490002 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.489977    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.490111 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.490010    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.490111 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.490037    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.490270 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.490115    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m47lc\" (UniqueName: \"kubernetes.io/projected/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-kube-api-access-m47lc\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.490270 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.490166    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.490270 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.490225    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.490390 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.490288    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.490390 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.490328    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.490390 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.490361    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.490390 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.490381    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.490511 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.490468    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.490550 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.490509    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.530703 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.530683    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:06.595166 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.591693    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.595166 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.591775    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.595166 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.591818    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.595166 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.591851    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.595166 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.591879    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m47lc\" (UniqueName: \"kubernetes.io/projected/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-kube-api-access-m47lc\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.595166 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.591914    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.595166 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.592985    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.595166 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.593172    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.595166 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.593884    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.595166 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.591940    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.595166 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.593978    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.595166 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.594009    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.595166 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.594067    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.595166 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.594100    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.595166 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.594154    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.598822 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.598794    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.599653 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.599624    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.599854 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.599833    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.600537 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.600511    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.601563 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.601540    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.601647 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.601590    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.601647 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.601040    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.603808 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.603788    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.618114 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.618093    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m47lc\" (UniqueName: \"kubernetes.io/projected/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-kube-api-access-m47lc\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:06.844743 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:06.844719    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:17:06.845795 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:17:06.845766    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400a4c20_1943_4e25_8816_2240d796b09b.slice/crio-a08ccc2817bf69cb7a4437fc1e22591798844b2cb292b4ede1bc45539f0db590 WatchSource:0}: Error finding container a08ccc2817bf69cb7a4437fc1e22591798844b2cb292b4ede1bc45539f0db590: Status 404 returned error can't find the container with id a08ccc2817bf69cb7a4437fc1e22591798844b2cb292b4ede1bc45539f0db590
Oct 30 04:17:07.786261 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:07.786220    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"400a4c20-1943-4e25-8816-2240d796b09b","Type":"ContainerStarted","Data":"39fd359b606a5367401563e47e548304ec7c2af37fe2b3f57b0a56e1ec76bd76"}
Oct 30 04:17:07.786637 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:07.786281    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"400a4c20-1943-4e25-8816-2240d796b09b","Type":"ContainerStarted","Data":"a08ccc2817bf69cb7a4437fc1e22591798844b2cb292b4ede1bc45539f0db590"}
Oct 30 04:17:11.805592 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:11.805557    4960 generic.go:356] "Generic (PLEG): container finished" podID="400a4c20-1943-4e25-8816-2240d796b09b" containerID="39fd359b606a5367401563e47e548304ec7c2af37fe2b3f57b0a56e1ec76bd76" exitCode=0
Oct 30 04:17:11.805962 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:11.805637    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"400a4c20-1943-4e25-8816-2240d796b09b","Type":"ContainerDied","Data":"39fd359b606a5367401563e47e548304ec7c2af37fe2b3f57b0a56e1ec76bd76"}
Oct 30 04:17:12.812637 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:12.812564    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"400a4c20-1943-4e25-8816-2240d796b09b","Type":"ContainerStarted","Data":"0722eb4a187cea1f806e897f3bbc8ec0d738da21ea281b1f41850037d0d34669"}
Oct 30 04:17:12.812637 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:12.812621    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"400a4c20-1943-4e25-8816-2240d796b09b","Type":"ContainerStarted","Data":"bee7509a7463f9d69c1db443e3b3920fdf2a850c312d4ccdbe2514beb532b648"}
Oct 30 04:17:12.813008 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:12.812803    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:12.887246 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:12.887206    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.302634419 podStartE2EDuration="6.887168565s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.584534146s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:12.872857165 +0000 UTC m=+5713.521905588" watchObservedRunningTime="2026-10-30 04:17:12.887168565 +0000 UTC m=+5713.536217051"
Oct 30 04:17:13.766144 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:13.766112    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"
Oct 30 04:17:14.751822 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:14.751790    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:14.784021 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:14.783975    4960 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:14.852595 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:14.852567    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:14.852718 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:14.852699    4960 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:14.868254 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:14.868233    4960 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") DevicePath \"csi-6f8c8e4c1326657251c30594036033694cadf25e97408e4f9d9da436f78f3fa4\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:14.972614 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:14.972590    4960 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/8c516923ea0339579ebf808cef81b9e0aaf55e5b619e2c6187025fa3b2b51f7f/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:15.007003 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:15.006945    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:15.079215 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:15.079171    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:15.403044 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:17:15.403004    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1a8ce5_ca44_4c12_b1f4_e5a522d516fe.slice/crio-b3eeebdc71008d71065c1cf46bbef6a0808fd126545add86a38b375131b2f68d WatchSource:0}: Error finding container b3eeebdc71008d71065c1cf46bbef6a0808fd126545add86a38b375131b2f68d: Status 404 returned error can't find the container with id b3eeebdc71008d71065c1cf46bbef6a0808fd126545add86a38b375131b2f68d
Oct 30 04:17:15.415824 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:15.415795    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:17:15.826395 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:15.826361    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe","Type":"ContainerStarted","Data":"66583d51a6aac90ded2eeb97865006b4856f5eb25374c9dd38db508196d62187"}
Oct 30 04:17:15.826703 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:15.826435    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe","Type":"ContainerStarted","Data":"b3eeebdc71008d71065c1cf46bbef6a0808fd126545add86a38b375131b2f68d"}
Oct 30 04:17:18.817878 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:18.817848    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:19.845695 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:19.845661    4960 generic.go:356] "Generic (PLEG): container finished" podID="ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" containerID="66583d51a6aac90ded2eeb97865006b4856f5eb25374c9dd38db508196d62187" exitCode=0
Oct 30 04:17:19.846056 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:19.845713    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe","Type":"ContainerDied","Data":"66583d51a6aac90ded2eeb97865006b4856f5eb25374c9dd38db508196d62187"}
Oct 30 04:17:20.853725 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:20.853693    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe","Type":"ContainerStarted","Data":"d98efc715606efcda1d0a4283ca4de9f33593bf41e0244491fe447bae2dd7c5a"}
Oct 30 04:17:20.854109 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:20.853755    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe","Type":"ContainerStarted","Data":"389f0df7dbbf2644feb0effe57283f915fad665751156900b3327e574978dc9d"}
Oct 30 04:17:20.854109 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:20.853780    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe","Type":"ContainerStarted","Data":"4e42a0c186018bc981be12ad27e4afc6e9468020c5afa7a2b220d50f7f906a76"}
Oct 30 04:17:20.930671 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:20.930621    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=11.546041202 podStartE2EDuration="14.930610444s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.384569242s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:20.916842344 +0000 UTC m=+5721.565890779" watchObservedRunningTime="2026-10-30 04:17:20.930610444 +0000 UTC m=+5721.579658909"
Oct 30 04:17:23.952658 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:17:23.952620    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:17:25.079642 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:25.079613    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:30.079906 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:30.079874    4960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:30.085049 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:30.085022    4960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:30.904765 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:30.904731    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:34.222404 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.222369    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"]
Oct 30 04:17:34.222944 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.222593    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q" podUID="8281b1f0-73c3-47c8-b82e-aabc93dfe3dd" containerName="korrel8r" containerID="cri-o://4451d6c8d9d0831f538aa93b989a1f6b6b399fd5178a55b7978cc08257cfe35e" gracePeriod=30
Oct 30 04:17:34.469062 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.469039    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"
Oct 30 04:17:34.607078 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.607050    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-tls\" (UniqueName: \"kubernetes.io/secret/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-tls\") pod \"8281b1f0-73c3-47c8-b82e-aabc93dfe3dd\" (UID: \"8281b1f0-73c3-47c8-b82e-aabc93dfe3dd\") "
Oct 30 04:17:34.607300 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.607089    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-kube-api-access-7bjjj\" (UniqueName: \"kubernetes.io/projected/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-kube-api-access-7bjjj\") pod \"8281b1f0-73c3-47c8-b82e-aabc93dfe3dd\" (UID: \"8281b1f0-73c3-47c8-b82e-aabc93dfe3dd\") "
Oct 30 04:17:34.607381 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.607324    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-config\" (UniqueName: \"kubernetes.io/configmap/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-config\") pod \"8281b1f0-73c3-47c8-b82e-aabc93dfe3dd\" (UID: \"8281b1f0-73c3-47c8-b82e-aabc93dfe3dd\") "
Oct 30 04:17:34.608477 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.608448    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-config" pod "8281b1f0-73c3-47c8-b82e-aabc93dfe3dd" (UID: "8281b1f0-73c3-47c8-b82e-aabc93dfe3dd"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:34.609758 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.609731    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-tls" pod "8281b1f0-73c3-47c8-b82e-aabc93dfe3dd" (UID: "8281b1f0-73c3-47c8-b82e-aabc93dfe3dd"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:34.609870 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.609810    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-kube-api-access-7bjjj" pod "8281b1f0-73c3-47c8-b82e-aabc93dfe3dd" (UID: "8281b1f0-73c3-47c8-b82e-aabc93dfe3dd"). InnerVolumeSpecName "kube-api-access-7bjjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:34.632028 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.632001    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:34.706699 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.706655    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:17:34.707030 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.706972    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" containerName="prometheus" containerID="cri-o://4e42a0c186018bc981be12ad27e4afc6e9468020c5afa7a2b220d50f7f906a76" gracePeriod=600
Oct 30 04:17:34.707172 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.707027    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" containerName="thanos-sidecar" containerID="cri-o://d98efc715606efcda1d0a4283ca4de9f33593bf41e0244491fe447bae2dd7c5a" gracePeriod=600
Oct 30 04:17:34.707172 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.707158    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" containerName="config-reloader" containerID="cri-o://389f0df7dbbf2644feb0effe57283f915fad665751156900b3327e574978dc9d" gracePeriod=600
Oct 30 04:17:34.709405 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.709383    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:34.709556 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.709535    4960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-tls\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:34.709687 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.709560    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bjjj\" (UniqueName: \"kubernetes.io/projected/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd-kube-api-access-7bjjj\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:34.717892 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.717863    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:17:34.718169 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.718143    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="400a4c20-1943-4e25-8816-2240d796b09b" containerName="alertmanager" containerID="cri-o://bee7509a7463f9d69c1db443e3b3920fdf2a850c312d4ccdbe2514beb532b648" gracePeriod=120
Oct 30 04:17:34.718362 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.718337    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="400a4c20-1943-4e25-8816-2240d796b09b" containerName="config-reloader" containerID="cri-o://0722eb4a187cea1f806e897f3bbc8ec0d738da21ea281b1f41850037d0d34669" gracePeriod=120
Oct 30 04:17:34.763898 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.763873    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:34.887313 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.887242    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:34.920840 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.920814    4960 generic.go:356] "Generic (PLEG): container finished" podID="8281b1f0-73c3-47c8-b82e-aabc93dfe3dd" containerID="4451d6c8d9d0831f538aa93b989a1f6b6b399fd5178a55b7978cc08257cfe35e" exitCode=0
Oct 30 04:17:34.920970 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.920845    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q" event={"ID":"8281b1f0-73c3-47c8-b82e-aabc93dfe3dd","Type":"ContainerDied","Data":"4451d6c8d9d0831f538aa93b989a1f6b6b399fd5178a55b7978cc08257cfe35e"}
Oct 30 04:17:34.920970 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.920863    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"
Oct 30 04:17:34.920970 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.920928    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q" event={"ID":"8281b1f0-73c3-47c8-b82e-aabc93dfe3dd","Type":"ContainerDied","Data":"c6d9c755084599ceac10d3d1a3f1d83afb6966a21b49b05e7577d0b3a2053010"}
Oct 30 04:17:34.920970 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.920941    4960 scope.go:122] "RemoveContainer" containerID="4451d6c8d9d0831f538aa93b989a1f6b6b399fd5178a55b7978cc08257cfe35e"
Oct 30 04:17:34.923800 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.923776    4960 generic.go:356] "Generic (PLEG): container finished" podID="400a4c20-1943-4e25-8816-2240d796b09b" containerID="0722eb4a187cea1f806e897f3bbc8ec0d738da21ea281b1f41850037d0d34669" exitCode=0
Oct 30 04:17:34.923905 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.923848    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"400a4c20-1943-4e25-8816-2240d796b09b","Type":"ContainerDied","Data":"0722eb4a187cea1f806e897f3bbc8ec0d738da21ea281b1f41850037d0d34669"}
Oct 30 04:17:34.926691 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.926673    4960 generic.go:356] "Generic (PLEG): container finished" podID="ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" containerID="d98efc715606efcda1d0a4283ca4de9f33593bf41e0244491fe447bae2dd7c5a" exitCode=0
Oct 30 04:17:34.926691 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.926689    4960 generic.go:356] "Generic (PLEG): container finished" podID="ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" containerID="389f0df7dbbf2644feb0effe57283f915fad665751156900b3327e574978dc9d" exitCode=0
Oct 30 04:17:34.926830 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.926695    4960 generic.go:356] "Generic (PLEG): container finished" podID="ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" containerID="4e42a0c186018bc981be12ad27e4afc6e9468020c5afa7a2b220d50f7f906a76" exitCode=0
Oct 30 04:17:34.926830 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.926715    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe","Type":"ContainerDied","Data":"d98efc715606efcda1d0a4283ca4de9f33593bf41e0244491fe447bae2dd7c5a"}
Oct 30 04:17:34.926830 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.926768    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe","Type":"ContainerDied","Data":"389f0df7dbbf2644feb0effe57283f915fad665751156900b3327e574978dc9d"}
Oct 30 04:17:34.926830 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.926793    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe","Type":"ContainerDied","Data":"4e42a0c186018bc981be12ad27e4afc6e9468020c5afa7a2b220d50f7f906a76"}
Oct 30 04:17:34.948510 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.948489    4960 scope.go:122] "RemoveContainer" containerID="4451d6c8d9d0831f538aa93b989a1f6b6b399fd5178a55b7978cc08257cfe35e"
Oct 30 04:17:34.948801 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:17:34.948783    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4451d6c8d9d0831f538aa93b989a1f6b6b399fd5178a55b7978cc08257cfe35e\": container with ID starting with 4451d6c8d9d0831f538aa93b989a1f6b6b399fd5178a55b7978cc08257cfe35e not found: ID does not exist" containerID="4451d6c8d9d0831f538aa93b989a1f6b6b399fd5178a55b7978cc08257cfe35e"
Oct 30 04:17:34.948857 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.948808    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4451d6c8d9d0831f538aa93b989a1f6b6b399fd5178a55b7978cc08257cfe35e"} err="failed to get container status \"4451d6c8d9d0831f538aa93b989a1f6b6b399fd5178a55b7978cc08257cfe35e\": rpc error: code = NotFound desc = could not find container \"4451d6c8d9d0831f538aa93b989a1f6b6b399fd5178a55b7978cc08257cfe35e\": container with ID starting with 4451d6c8d9d0831f538aa93b989a1f6b6b399fd5178a55b7978cc08257cfe35e not found: ID does not exist"
Oct 30 04:17:34.984998 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:34.984977    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"]
Oct 30 04:17:35.005382 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.005359    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gl24q"]
Oct 30 04:17:35.039106 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.039076    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:35.354895 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.349694    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:17:35.354895 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.349984    4960 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="f943646f-688d-41b9-9b16-4d54b4195bc2" containerName="otc-container" containerID="cri-o://c887ea279eb15539b3d79ac80cccc0da79c341bf2f12211dfef6cb8d4bf87568" gracePeriod=30
Oct 30 04:17:35.407158 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.407112    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:35.504009 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.503911    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8281b1f0-73c3-47c8-b82e-aabc93dfe3dd" path="/var/lib/kubelet/pods/8281b1f0-73c3-47c8-b82e-aabc93dfe3dd/volumes"
Oct 30 04:17:35.532442 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.532417    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:35.541510 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.541488    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:35.611023 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.610996    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:35.617832 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.617811    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") "
Oct 30 04:17:35.617950 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.617849    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") "
Oct 30 04:17:35.617950 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.617869    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-kube-api-access-m47lc\" (UniqueName: \"kubernetes.io/projected/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-kube-api-access-m47lc\") pod \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") "
Oct 30 04:17:35.617950 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.617894    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config\") pod \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") "
Oct 30 04:17:35.617950 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.617914    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-tls-assets\" (UniqueName: \"kubernetes.io/projected/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-tls-assets\") pod \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") "
Oct 30 04:17:35.617950 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.617933    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-thanos-prometheus-http-client-file\") pod \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") "
Oct 30 04:17:35.619653 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.618466    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") "
Oct 30 04:17:35.619653 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.618483    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-2" pod "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" (UID: "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:35.619653 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.618615    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-config-out\") pod \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") "
Oct 30 04:17:35.619653 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.618766    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-secret-prometheus-web-tls-ca\") pod \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") "
Oct 30 04:17:35.619653 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.618810    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-config\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-config\") pod \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") "
Oct 30 04:17:35.619653 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.618878    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-1" pod "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" (UID: "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:35.619653 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.618896    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") "
Oct 30 04:17:35.619653 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.618945    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") "
Oct 30 04:17:35.619653 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.619064    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\" (UID: \"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe\") "
Oct 30 04:17:35.619653 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.619355    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.619653 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.619376    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.619653 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.619581    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-0" pod "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" (UID: "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:35.621736 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.621710    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-kube-api-access-m47lc" pod "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" (UID: "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe"). InnerVolumeSpecName "kube-api-access-m47lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:35.623016 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.622985    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" (UID: "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.623525 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.623505    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-config-out" pod "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" (UID: "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:35.623781 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.623534    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" (UID: "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.623781 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.623629    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-config" pod "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" (UID: "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.624006 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.623980    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-secret-prometheus-web-tls-ca" pod "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" (UID: "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.627582 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.627558    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-thanos-prometheus-http-client-file" pod "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" (UID: "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.627746 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.627725    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87" pod "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" (UID: "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe"). InnerVolumeSpecName "pvc-9689c530-7f91-4584-b28e-8809bed9e799". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:17:35.629057 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.629037    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-tls-assets" pod "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" (UID: "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:35.637507 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.637483    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config" pod "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" (UID: "ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.651646 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.651620    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:35.720232 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.720201    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-monitoring-tls-certs\") pod \"f943646f-688d-41b9-9b16-4d54b4195bc2\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") "
Oct 30 04:17:35.720438 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.720290    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f943646f-688d-41b9-9b16-4d54b4195bc2-otc-internal\" (UniqueName: \"kubernetes.io/configmap/f943646f-688d-41b9-9b16-4d54b4195bc2-otc-internal\") pod \"f943646f-688d-41b9-9b16-4d54b4195bc2\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") "
Oct 30 04:17:35.720438 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.720325    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-tls-certs\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-tls-certs\") pod \"f943646f-688d-41b9-9b16-4d54b4195bc2\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") "
Oct 30 04:17:35.720438 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.720356    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f943646f-688d-41b9-9b16-4d54b4195bc2-kube-api-access-f7wqt\" (UniqueName: \"kubernetes.io/projected/f943646f-688d-41b9-9b16-4d54b4195bc2-kube-api-access-f7wqt\") pod \"f943646f-688d-41b9-9b16-4d54b4195bc2\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") "
Oct 30 04:17:35.720438 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.720398    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-otlp-tls-certs\") pod \"f943646f-688d-41b9-9b16-4d54b4195bc2\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") "
Oct 30 04:17:35.722812 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.720966    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-data-science-collector-ta-client-cert\") pod \"f943646f-688d-41b9-9b16-4d54b4195bc2\" (UID: \"f943646f-688d-41b9-9b16-4d54b4195bc2\") "
Oct 30 04:17:35.722812 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.721255    4960 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.722812 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.721278    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m47lc\" (UniqueName: \"kubernetes.io/projected/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-kube-api-access-m47lc\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.722812 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.721294    4960 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.722812 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.721307    4960 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-tls-assets\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.722812 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.721320    4960 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.722812 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.721336    4960 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-config-out\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.722812 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.721350    4960 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-secret-prometheus-web-tls-ca\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.722812 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.721363    4960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.722812 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.721388    4960 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") on node \"ip-10-0-128-241.ec2.internal\" "
Oct 30 04:17:35.722812 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.721406    4960 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.722812 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.721452    4960 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.722812 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.721670    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f943646f-688d-41b9-9b16-4d54b4195bc2-otc-internal" pod "f943646f-688d-41b9-9b16-4d54b4195bc2" (UID: "f943646f-688d-41b9-9b16-4d54b4195bc2"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:35.723500 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.723475    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-monitoring-tls-certs" pod "f943646f-688d-41b9-9b16-4d54b4195bc2" (UID: "f943646f-688d-41b9-9b16-4d54b4195bc2"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.723958 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.723935    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-tls-certs" pod "f943646f-688d-41b9-9b16-4d54b4195bc2" (UID: "f943646f-688d-41b9-9b16-4d54b4195bc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.724348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.724328    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f943646f-688d-41b9-9b16-4d54b4195bc2-kube-api-access-f7wqt" pod "f943646f-688d-41b9-9b16-4d54b4195bc2" (UID: "f943646f-688d-41b9-9b16-4d54b4195bc2"). InnerVolumeSpecName "kube-api-access-f7wqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:35.724469 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.724449    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-otlp-tls-certs" pod "f943646f-688d-41b9-9b16-4d54b4195bc2" (UID: "f943646f-688d-41b9-9b16-4d54b4195bc2"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.724772 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.724752    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-data-science-collector-ta-client-cert" pod "f943646f-688d-41b9-9b16-4d54b4195bc2" (UID: "f943646f-688d-41b9-9b16-4d54b4195bc2"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.736248 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.736233    4960 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-9689c530-7f91-4584-b28e-8809bed9e799" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87") on node "ip-10-0-128-241.ec2.internal"
Oct 30 04:17:35.821903 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.821884    4960 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f943646f-688d-41b9-9b16-4d54b4195bc2-otc-internal\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.822020 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.821905    4960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.822020 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.821918    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7wqt\" (UniqueName: \"kubernetes.io/projected/f943646f-688d-41b9-9b16-4d54b4195bc2-kube-api-access-f7wqt\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.822020 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.821932    4960 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-otlp-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.822020 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.821949    4960 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-data-science-collector-ta-client-cert\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.822020 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.821964    4960 reconciler_common.go:299] "Volume detached for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"csi-6f8c8e4c1326657251c30594036033694cadf25e97408e4f9d9da436f78f3fa4\""
Oct 30 04:17:35.822020 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.821979    4960 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f943646f-688d-41b9-9b16-4d54b4195bc2-monitoring-tls-certs\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.933865 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.933835    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:35.933865 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.933843    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe","Type":"ContainerDied","Data":"b3eeebdc71008d71065c1cf46bbef6a0808fd126545add86a38b375131b2f68d"}
Oct 30 04:17:35.934068 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.933923    4960 scope.go:122] "RemoveContainer" containerID="d98efc715606efcda1d0a4283ca4de9f33593bf41e0244491fe447bae2dd7c5a"
Oct 30 04:17:35.935469 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.935442    4960 generic.go:356] "Generic (PLEG): container finished" podID="f943646f-688d-41b9-9b16-4d54b4195bc2" containerID="c887ea279eb15539b3d79ac80cccc0da79c341bf2f12211dfef6cb8d4bf87568" exitCode=0
Oct 30 04:17:35.935735 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.935470    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f943646f-688d-41b9-9b16-4d54b4195bc2","Type":"ContainerDied","Data":"c887ea279eb15539b3d79ac80cccc0da79c341bf2f12211dfef6cb8d4bf87568"}
Oct 30 04:17:35.935735 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.935498    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:17:35.935735 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.935523    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f943646f-688d-41b9-9b16-4d54b4195bc2","Type":"ContainerDied","Data":"40b16023d827a87c67490928f77a34100e0a6d31bc01c826d14369127e36ddf5"}
Oct 30 04:17:35.944320 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.944293    4960 scope.go:122] "RemoveContainer" containerID="389f0df7dbbf2644feb0effe57283f915fad665751156900b3327e574978dc9d"
Oct 30 04:17:35.952504 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.952486    4960 scope.go:122] "RemoveContainer" containerID="4e42a0c186018bc981be12ad27e4afc6e9468020c5afa7a2b220d50f7f906a76"
Oct 30 04:17:35.960646 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.960629    4960 scope.go:122] "RemoveContainer" containerID="66583d51a6aac90ded2eeb97865006b4856f5eb25374c9dd38db508196d62187"
Oct 30 04:17:35.968132 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.968114    4960 scope.go:122] "RemoveContainer" containerID="c887ea279eb15539b3d79ac80cccc0da79c341bf2f12211dfef6cb8d4bf87568"
Oct 30 04:17:35.975529 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.975512    4960 scope.go:122] "RemoveContainer" containerID="c887ea279eb15539b3d79ac80cccc0da79c341bf2f12211dfef6cb8d4bf87568"
Oct 30 04:17:35.975792 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:17:35.975773    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c887ea279eb15539b3d79ac80cccc0da79c341bf2f12211dfef6cb8d4bf87568\": container with ID starting with c887ea279eb15539b3d79ac80cccc0da79c341bf2f12211dfef6cb8d4bf87568 not found: ID does not exist" containerID="c887ea279eb15539b3d79ac80cccc0da79c341bf2f12211dfef6cb8d4bf87568"
Oct 30 04:17:35.975855 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:35.975802    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c887ea279eb15539b3d79ac80cccc0da79c341bf2f12211dfef6cb8d4bf87568"} err="failed to get container status \"c887ea279eb15539b3d79ac80cccc0da79c341bf2f12211dfef6cb8d4bf87568\": rpc error: code = NotFound desc = could not find container \"c887ea279eb15539b3d79ac80cccc0da79c341bf2f12211dfef6cb8d4bf87568\": container with ID starting with c887ea279eb15539b3d79ac80cccc0da79c341bf2f12211dfef6cb8d4bf87568 not found: ID does not exist"
Oct 30 04:17:36.034364 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:36.034343    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:17:36.048089 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:36.048069    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:17:36.085362 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:36.085315    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:17:36.097426 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:36.097402    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:17:36.522034 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:36.521961    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:36.640786 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:36.640757    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:36.761768 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:36.761737    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:37.502335 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:37.502298    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" path="/var/lib/kubelet/pods/ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe/volumes"
Oct 30 04:17:37.503028 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:37.503005    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f943646f-688d-41b9-9b16-4d54b4195bc2" path="/var/lib/kubelet/pods/f943646f-688d-41b9-9b16-4d54b4195bc2/volumes"
Oct 30 04:17:38.207945 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:38.207913    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:38.312035 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:38.312006    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:38.492814 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:38.492739    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:38.609533 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:38.609504    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:38.817933 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:38.817903    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="400a4c20-1943-4e25-8816-2240d796b09b" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.140:9093/-/ready\": dial tcp 10.132.0.140:9093: connect: connection refused"
Oct 30 04:17:43.817964 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:43.817907    4960 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="400a4c20-1943-4e25-8816-2240d796b09b" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.140:9093/-/ready\": dial tcp 10.132.0.140:9093: connect: connection refused"
Oct 30 04:17:44.964350 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:44.964324    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:44.982711 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:44.982654    4960 generic.go:356] "Generic (PLEG): container finished" podID="400a4c20-1943-4e25-8816-2240d796b09b" containerID="bee7509a7463f9d69c1db443e3b3920fdf2a850c312d4ccdbe2514beb532b648" exitCode=0
Oct 30 04:17:44.982711 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:44.982700    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"400a4c20-1943-4e25-8816-2240d796b09b","Type":"ContainerDied","Data":"bee7509a7463f9d69c1db443e3b3920fdf2a850c312d4ccdbe2514beb532b648"}
Oct 30 04:17:44.982900 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:44.982715    4960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 04:17:44.982900 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:44.982776    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"400a4c20-1943-4e25-8816-2240d796b09b","Type":"ContainerDied","Data":"a08ccc2817bf69cb7a4437fc1e22591798844b2cb292b4ede1bc45539f0db590"}
Oct 30 04:17:44.982900 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:44.982807    4960 scope.go:122] "RemoveContainer" containerID="0722eb4a187cea1f806e897f3bbc8ec0d738da21ea281b1f41850037d0d34669"
Oct 30 04:17:44.990693 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:44.990676    4960 scope.go:122] "RemoveContainer" containerID="bee7509a7463f9d69c1db443e3b3920fdf2a850c312d4ccdbe2514beb532b648"
Oct 30 04:17:44.998039 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:44.998012    4960 scope.go:122] "RemoveContainer" containerID="39fd359b606a5367401563e47e548304ec7c2af37fe2b3f57b0a56e1ec76bd76"
Oct 30 04:17:45.008092 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.008075    4960 scope.go:122] "RemoveContainer" containerID="0722eb4a187cea1f806e897f3bbc8ec0d738da21ea281b1f41850037d0d34669"
Oct 30 04:17:45.008389 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:17:45.008370    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0722eb4a187cea1f806e897f3bbc8ec0d738da21ea281b1f41850037d0d34669\": container with ID starting with 0722eb4a187cea1f806e897f3bbc8ec0d738da21ea281b1f41850037d0d34669 not found: ID does not exist" containerID="0722eb4a187cea1f806e897f3bbc8ec0d738da21ea281b1f41850037d0d34669"
Oct 30 04:17:45.008467 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.008394    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0722eb4a187cea1f806e897f3bbc8ec0d738da21ea281b1f41850037d0d34669"} err="failed to get container status \"0722eb4a187cea1f806e897f3bbc8ec0d738da21ea281b1f41850037d0d34669\": rpc error: code = NotFound desc = could not find container \"0722eb4a187cea1f806e897f3bbc8ec0d738da21ea281b1f41850037d0d34669\": container with ID starting with 0722eb4a187cea1f806e897f3bbc8ec0d738da21ea281b1f41850037d0d34669 not found: ID does not exist"
Oct 30 04:17:45.008467 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.008411    4960 scope.go:122] "RemoveContainer" containerID="bee7509a7463f9d69c1db443e3b3920fdf2a850c312d4ccdbe2514beb532b648"
Oct 30 04:17:45.008693 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:17:45.008676    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bee7509a7463f9d69c1db443e3b3920fdf2a850c312d4ccdbe2514beb532b648\": container with ID starting with bee7509a7463f9d69c1db443e3b3920fdf2a850c312d4ccdbe2514beb532b648 not found: ID does not exist" containerID="bee7509a7463f9d69c1db443e3b3920fdf2a850c312d4ccdbe2514beb532b648"
Oct 30 04:17:45.008781 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.008697    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bee7509a7463f9d69c1db443e3b3920fdf2a850c312d4ccdbe2514beb532b648"} err="failed to get container status \"bee7509a7463f9d69c1db443e3b3920fdf2a850c312d4ccdbe2514beb532b648\": rpc error: code = NotFound desc = could not find container \"bee7509a7463f9d69c1db443e3b3920fdf2a850c312d4ccdbe2514beb532b648\": container with ID starting with bee7509a7463f9d69c1db443e3b3920fdf2a850c312d4ccdbe2514beb532b648 not found: ID does not exist"
Oct 30 04:17:45.008781 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.008713    4960 scope.go:122] "RemoveContainer" containerID="39fd359b606a5367401563e47e548304ec7c2af37fe2b3f57b0a56e1ec76bd76"
Oct 30 04:17:45.008998 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:17:45.008979    4960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39fd359b606a5367401563e47e548304ec7c2af37fe2b3f57b0a56e1ec76bd76\": container with ID starting with 39fd359b606a5367401563e47e548304ec7c2af37fe2b3f57b0a56e1ec76bd76 not found: ID does not exist" containerID="39fd359b606a5367401563e47e548304ec7c2af37fe2b3f57b0a56e1ec76bd76"
Oct 30 04:17:45.009038 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.009009    4960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39fd359b606a5367401563e47e548304ec7c2af37fe2b3f57b0a56e1ec76bd76"} err="failed to get container status \"39fd359b606a5367401563e47e548304ec7c2af37fe2b3f57b0a56e1ec76bd76\": rpc error: code = NotFound desc = could not find container \"39fd359b606a5367401563e47e548304ec7c2af37fe2b3f57b0a56e1ec76bd76\": container with ID starting with 39fd359b606a5367401563e47e548304ec7c2af37fe2b3f57b0a56e1ec76bd76 not found: ID does not exist"
Oct 30 04:17:45.098516 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.098494    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/400a4c20-1943-4e25-8816-2240d796b09b-tls-assets\" (UniqueName: \"kubernetes.io/projected/400a4c20-1943-4e25-8816-2240d796b09b-tls-assets\") pod \"400a4c20-1943-4e25-8816-2240d796b09b\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") "
Oct 30 04:17:45.098615 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.098530    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/400a4c20-1943-4e25-8816-2240d796b09b-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/400a4c20-1943-4e25-8816-2240d796b09b-alertmanager-data-science-monitoringstack-db\") pod \"400a4c20-1943-4e25-8816-2240d796b09b\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") "
Oct 30 04:17:45.098615 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.098582    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/400a4c20-1943-4e25-8816-2240d796b09b-kube-api-access-7hgcw\" (UniqueName: \"kubernetes.io/projected/400a4c20-1943-4e25-8816-2240d796b09b-kube-api-access-7hgcw\") pod \"400a4c20-1943-4e25-8816-2240d796b09b\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") "
Oct 30 04:17:45.098615 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.098610    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/400a4c20-1943-4e25-8816-2240d796b09b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/400a4c20-1943-4e25-8816-2240d796b09b-config-out\") pod \"400a4c20-1943-4e25-8816-2240d796b09b\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") "
Oct 30 04:17:45.098771 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.098660    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-config-volume\" (UniqueName: \"kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-config-volume\") pod \"400a4c20-1943-4e25-8816-2240d796b09b\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") "
Oct 30 04:17:45.098771 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.098687    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-web-config\" (UniqueName: \"kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-web-config\") pod \"400a4c20-1943-4e25-8816-2240d796b09b\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") "
Oct 30 04:17:45.098771 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.098764    4960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-cluster-tls-config\") pod \"400a4c20-1943-4e25-8816-2240d796b09b\" (UID: \"400a4c20-1943-4e25-8816-2240d796b09b\") "
Oct 30 04:17:45.098969 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.098942    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400a4c20-1943-4e25-8816-2240d796b09b-alertmanager-data-science-monitoringstack-db" pod "400a4c20-1943-4e25-8816-2240d796b09b" (UID: "400a4c20-1943-4e25-8816-2240d796b09b"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:45.099485 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.099454    4960 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/400a4c20-1943-4e25-8816-2240d796b09b-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:45.100929 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.100901    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400a4c20-1943-4e25-8816-2240d796b09b-tls-assets" pod "400a4c20-1943-4e25-8816-2240d796b09b" (UID: "400a4c20-1943-4e25-8816-2240d796b09b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:45.101152 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.101130    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-config-volume" pod "400a4c20-1943-4e25-8816-2240d796b09b" (UID: "400a4c20-1943-4e25-8816-2240d796b09b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:45.101389 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.101370    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400a4c20-1943-4e25-8816-2240d796b09b-config-out" pod "400a4c20-1943-4e25-8816-2240d796b09b" (UID: "400a4c20-1943-4e25-8816-2240d796b09b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:45.101486 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.101462    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400a4c20-1943-4e25-8816-2240d796b09b-kube-api-access-7hgcw" pod "400a4c20-1943-4e25-8816-2240d796b09b" (UID: "400a4c20-1943-4e25-8816-2240d796b09b"). InnerVolumeSpecName "kube-api-access-7hgcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:45.105196 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.105156    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-cluster-tls-config" pod "400a4c20-1943-4e25-8816-2240d796b09b" (UID: "400a4c20-1943-4e25-8816-2240d796b09b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:45.112329 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.112309    4960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-web-config" pod "400a4c20-1943-4e25-8816-2240d796b09b" (UID: "400a4c20-1943-4e25-8816-2240d796b09b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:45.200713 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.200695    4960 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-web-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:45.200713 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.200716    4960 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-cluster-tls-config\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:45.200847 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.200725    4960 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/400a4c20-1943-4e25-8816-2240d796b09b-tls-assets\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:45.200847 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.200733    4960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hgcw\" (UniqueName: \"kubernetes.io/projected/400a4c20-1943-4e25-8816-2240d796b09b-kube-api-access-7hgcw\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:45.200847 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.200743    4960 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/400a4c20-1943-4e25-8816-2240d796b09b-config-out\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:45.200847 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.200751    4960 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/400a4c20-1943-4e25-8816-2240d796b09b-config-volume\") on node \"ip-10-0-128-241.ec2.internal\" DevicePath \"\""
Oct 30 04:17:45.333634 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.333604    4960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:17:45.341020 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.340992    4960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 04:17:45.503827 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:45.503799    4960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="400a4c20-1943-4e25-8816-2240d796b09b" path="/var/lib/kubelet/pods/400a4c20-1943-4e25-8816-2240d796b09b/volumes"
Oct 30 04:17:53.955202 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:17:53.955139    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:17:59.586801 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:17:59.586763    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:00.526532 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:00.526505    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:05.291216 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:05.291166    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:05.400157 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:05.400125    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:09.629585 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:09.629554    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-observability-system_odh-observability-5db8f88448-5w2qq_62e56e71-7569-404d-a541-3d8520aea56c/manager/0.log"
Oct 30 04:18:10.233325 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:10.233294    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:10.351238 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:10.351205    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:14.752511 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:14.752477    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-skc8q_8a3ebba8-b807-4256-b552-5a18ec8412cb/global-pull-secret-syncer/0.log"
Oct 30 04:18:14.896399 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:14.896371    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-2sq5m_b1337da2-13ee-4b79-9a84-37ef217ae10f/connection-checker/0.log"
Oct 30 04:18:15.005301 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:15.005230    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-dhf78_c1cf080d-9f7d-4614-b446-76d174d9c3f7/connection-checker/0.log"
Oct 30 04:18:15.111968 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:15.111938    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jsbwd_0d2d9d6b-2705-49d3-b3d2-079d6d85148f/connection-checker/0.log"
Oct 30 04:18:15.247295 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:15.247271    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log"
Oct 30 04:18:15.849276 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:15.849246    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-241.ec2.internal_2ea871c29059b568921cda32c2dd3ba5/haproxy/0.log"
Oct 30 04:18:21.587078 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:21.587040    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-vjvcf_d6516df6-4df4-4213-890a-0b25e274e606/cluster-monitoring-operator/0.log"
Oct 30 04:18:22.739997 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:22.739968    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c8988_b94b540a-e972-41a6-8b11-e50b428e6f75/node-exporter/0.log"
Oct 30 04:18:22.847373 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:22.847348    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c8988_b94b540a-e972-41a6-8b11-e50b428e6f75/kube-rbac-proxy/0.log"
Oct 30 04:18:22.956257 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:22.956233    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c8988_b94b540a-e972-41a6-8b11-e50b428e6f75/init-textfile/0.log"
Oct 30 04:18:23.430382 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.430343    4960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"]
Oct 30 04:18:23.430752 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.430731    4960 cpu_manager.go:414] "containerMap: removing container" podUID="ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" containerName="thanos-sidecar"
Oct 30 04:18:23.430850 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.430754    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" containerName="thanos-sidecar"
Oct 30 04:18:23.430850 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.430780    4960 cpu_manager.go:414] "containerMap: removing container" podUID="ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" containerName="config-reloader"
Oct 30 04:18:23.430850 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.430787    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" containerName="config-reloader"
Oct 30 04:18:23.430850 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.430802    4960 cpu_manager.go:414] "containerMap: removing container" podUID="8281b1f0-73c3-47c8-b82e-aabc93dfe3dd" containerName="korrel8r"
Oct 30 04:18:23.430850 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.430809    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8281b1f0-73c3-47c8-b82e-aabc93dfe3dd" containerName="korrel8r"
Oct 30 04:18:23.430850 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.430820    4960 cpu_manager.go:414] "containerMap: removing container" podUID="ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" containerName="init-config-reloader"
Oct 30 04:18:23.430850 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.430827    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" containerName="init-config-reloader"
Oct 30 04:18:23.430850 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.430839    4960 cpu_manager.go:414] "containerMap: removing container" podUID="400a4c20-1943-4e25-8816-2240d796b09b" containerName="alertmanager"
Oct 30 04:18:23.430850 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.430846    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400a4c20-1943-4e25-8816-2240d796b09b" containerName="alertmanager"
Oct 30 04:18:23.431348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.430858    4960 cpu_manager.go:414] "containerMap: removing container" podUID="400a4c20-1943-4e25-8816-2240d796b09b" containerName="config-reloader"
Oct 30 04:18:23.431348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.430865    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400a4c20-1943-4e25-8816-2240d796b09b" containerName="config-reloader"
Oct 30 04:18:23.431348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.430874    4960 cpu_manager.go:414] "containerMap: removing container" podUID="ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" containerName="prometheus"
Oct 30 04:18:23.431348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.430883    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" containerName="prometheus"
Oct 30 04:18:23.431348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.430905    4960 cpu_manager.go:414] "containerMap: removing container" podUID="f943646f-688d-41b9-9b16-4d54b4195bc2" containerName="otc-container"
Oct 30 04:18:23.431348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.430914    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f943646f-688d-41b9-9b16-4d54b4195bc2" containerName="otc-container"
Oct 30 04:18:23.431348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.430940    4960 cpu_manager.go:414] "containerMap: removing container" podUID="400a4c20-1943-4e25-8816-2240d796b09b" containerName="init-config-reloader"
Oct 30 04:18:23.431348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.430947    4960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400a4c20-1943-4e25-8816-2240d796b09b" containerName="init-config-reloader"
Oct 30 04:18:23.431348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.431035    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" containerName="thanos-sidecar"
Oct 30 04:18:23.431348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.431051    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" containerName="config-reloader"
Oct 30 04:18:23.431348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.431061    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="8281b1f0-73c3-47c8-b82e-aabc93dfe3dd" containerName="korrel8r"
Oct 30 04:18:23.431348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.431069    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="400a4c20-1943-4e25-8816-2240d796b09b" containerName="alertmanager"
Oct 30 04:18:23.431348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.431078    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="400a4c20-1943-4e25-8816-2240d796b09b" containerName="config-reloader"
Oct 30 04:18:23.431348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.431087    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1a8ce5-ca44-4c12-b1f4-e5a522d516fe" containerName="prometheus"
Oct 30 04:18:23.431348 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.431097    4960 memory_manager.go:365] "RemoveStaleState removing state" podUID="f943646f-688d-41b9-9b16-4d54b4195bc2" containerName="otc-container"
Oct 30 04:18:23.434218 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.434196    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"
Oct 30 04:18:23.449964 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.449938    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6xn7p\"/\"kube-root-ca.crt\""
Oct 30 04:18:23.450055 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.449974    4960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6xn7p\"/\"openshift-service-ca.crt\""
Oct 30 04:18:23.463637 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.463618    4960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6xn7p\"/\"default-dockercfg-4j88g\""
Oct 30 04:18:23.464729 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.464706    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"]
Oct 30 04:18:23.465839 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.465820    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28a80c42-17b3-4001-97c4-dc4126125960-proc\") pod \"perf-node-gather-daemonset-dv282\" (UID: \"28a80c42-17b3-4001-97c4-dc4126125960\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"
Oct 30 04:18:23.465911 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.465858    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28a80c42-17b3-4001-97c4-dc4126125960-podres\") pod \"perf-node-gather-daemonset-dv282\" (UID: \"28a80c42-17b3-4001-97c4-dc4126125960\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"
Oct 30 04:18:23.465911 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.465891    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dmzl\" (UniqueName: \"kubernetes.io/projected/28a80c42-17b3-4001-97c4-dc4126125960-kube-api-access-9dmzl\") pod \"perf-node-gather-daemonset-dv282\" (UID: \"28a80c42-17b3-4001-97c4-dc4126125960\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"
Oct 30 04:18:23.465998 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.465980    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28a80c42-17b3-4001-97c4-dc4126125960-sys\") pod \"perf-node-gather-daemonset-dv282\" (UID: \"28a80c42-17b3-4001-97c4-dc4126125960\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"
Oct 30 04:18:23.466036 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.466008    4960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28a80c42-17b3-4001-97c4-dc4126125960-lib-modules\") pod \"perf-node-gather-daemonset-dv282\" (UID: \"28a80c42-17b3-4001-97c4-dc4126125960\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"
Oct 30 04:18:23.567243 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.567208    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28a80c42-17b3-4001-97c4-dc4126125960-podres\") pod \"perf-node-gather-daemonset-dv282\" (UID: \"28a80c42-17b3-4001-97c4-dc4126125960\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"
Oct 30 04:18:23.567353 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.567260    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dmzl\" (UniqueName: \"kubernetes.io/projected/28a80c42-17b3-4001-97c4-dc4126125960-kube-api-access-9dmzl\") pod \"perf-node-gather-daemonset-dv282\" (UID: \"28a80c42-17b3-4001-97c4-dc4126125960\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"
Oct 30 04:18:23.567353 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.567284    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28a80c42-17b3-4001-97c4-dc4126125960-sys\") pod \"perf-node-gather-daemonset-dv282\" (UID: \"28a80c42-17b3-4001-97c4-dc4126125960\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"
Oct 30 04:18:23.567353 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.567300    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28a80c42-17b3-4001-97c4-dc4126125960-lib-modules\") pod \"perf-node-gather-daemonset-dv282\" (UID: \"28a80c42-17b3-4001-97c4-dc4126125960\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"
Oct 30 04:18:23.567353 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.567327    4960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28a80c42-17b3-4001-97c4-dc4126125960-proc\") pod \"perf-node-gather-daemonset-dv282\" (UID: \"28a80c42-17b3-4001-97c4-dc4126125960\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"
Oct 30 04:18:23.567353 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.567329    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28a80c42-17b3-4001-97c4-dc4126125960-podres\") pod \"perf-node-gather-daemonset-dv282\" (UID: \"28a80c42-17b3-4001-97c4-dc4126125960\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"
Oct 30 04:18:23.567541 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.567452    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28a80c42-17b3-4001-97c4-dc4126125960-sys\") pod \"perf-node-gather-daemonset-dv282\" (UID: \"28a80c42-17b3-4001-97c4-dc4126125960\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"
Oct 30 04:18:23.567541 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.567477    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28a80c42-17b3-4001-97c4-dc4126125960-proc\") pod \"perf-node-gather-daemonset-dv282\" (UID: \"28a80c42-17b3-4001-97c4-dc4126125960\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"
Oct 30 04:18:23.567541 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.567494    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28a80c42-17b3-4001-97c4-dc4126125960-lib-modules\") pod \"perf-node-gather-daemonset-dv282\" (UID: \"28a80c42-17b3-4001-97c4-dc4126125960\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"
Oct 30 04:18:23.587947 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.587923    4960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dmzl\" (UniqueName: \"kubernetes.io/projected/28a80c42-17b3-4001-97c4-dc4126125960-kube-api-access-9dmzl\") pod \"perf-node-gather-daemonset-dv282\" (UID: \"28a80c42-17b3-4001-97c4-dc4126125960\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"
Oct 30 04:18:23.745924 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:23.745874    4960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"
Oct 30 04:18:23.952420 ip-10-0-128-241 kubenswrapper[4960]: E1030 04:18:23.952388    4960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dkfnj" containerName="extractor"
Oct 30 04:18:24.034461 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:24.034439    4960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"]
Oct 30 04:18:24.036331 ip-10-0-128-241 kubenswrapper[4960]: W1030 04:18:24.036297    4960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod28a80c42_17b3_4001_97c4_dc4126125960.slice/crio-99df282188e9d628918f6d6878e2ed40a4a7fd495e2a90c6e4a19b628e44b357 WatchSource:0}: Error finding container 99df282188e9d628918f6d6878e2ed40a4a7fd495e2a90c6e4a19b628e44b357: Status 404 returned error can't find the container with id 99df282188e9d628918f6d6878e2ed40a4a7fd495e2a90c6e4a19b628e44b357
Oct 30 04:18:24.170066 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:24.170025    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282" event={"ID":"28a80c42-17b3-4001-97c4-dc4126125960","Type":"ContainerStarted","Data":"7b164d7a3def516f0ba01138b8dddbae7a1453572ddb67adb033ef0d1e2e0a8c"}
Oct 30 04:18:24.170212 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:24.170089    4960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282" event={"ID":"28a80c42-17b3-4001-97c4-dc4126125960","Type":"ContainerStarted","Data":"99df282188e9d628918f6d6878e2ed40a4a7fd495e2a90c6e4a19b628e44b357"}
Oct 30 04:18:24.170212 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:24.170129    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"
Oct 30 04:18:24.211036 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:24.210989    4960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282" podStartSLOduration=1.2109773 podStartE2EDuration="1.2109773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:24.209930913 +0000 UTC m=+5784.858979334" watchObservedRunningTime="2026-10-30 04:18:24.2109773 +0000 UTC m=+5784.860025721"
Oct 30 04:18:27.241515 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:27.241487    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-jwzx9_70f081e1-7202-4b86-930d-8c3f72a3bb7e/networking-console-plugin/0.log"
Oct 30 04:18:28.204085 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:28.204052    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/2.log"
Oct 30 04:18:28.208447 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:28.208422    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-scstg_fb37ddc9-9a43-46c5-90e5-e8244f9dfe9e/console-operator/3.log"
Oct 30 04:18:28.264388 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:28.264365    4960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xpbx_d28e48a8-f70c-4949-b407-4c0e632f6a35/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:30.185021 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:30.184994    4960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-dv282"
Oct 30 04:18:30.246226 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:30.246199    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-njw6d_7e8de64a-fd25-4aa6-9160-b25e09d116aa/volume-data-source-validator/0.log"
Oct 30 04:18:32.237824 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:32.237800    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7mhtz_c5d42a36-81a2-49ad-9fa0-32d42e87d75d/dns/0.log"
Oct 30 04:18:32.345336 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:32.345310    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7mhtz_c5d42a36-81a2-49ad-9fa0-32d42e87d75d/kube-rbac-proxy/0.log"
Oct 30 04:18:32.935889 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:32.935865    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6pzrk_38e84dd1-8ee0-4a74-8802-3e3200c96642/dns-node-resolver/0.log"
Oct 30 04:18:34.627914 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:34.627888    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tvbs5_a2796ed5-6fd6-4231-9847-1bee8c9f9dd7/node-ca/0.log"
Oct 30 04:18:36.353952 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:36.353919    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5d4bdb4f7f-89p4x_cb0c5682-acc6-4029-831d-b6aa9027db25/router/0.log"
Oct 30 04:18:37.326276 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:37.326241    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fx6bk_f697930c-fefc-42f2-835e-7b41cf115aaf/serve-healthcheck-canary/0.log"
Oct 30 04:18:38.298678 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:38.298645    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-gxb72_f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9/insights-operator/0.log"
Oct 30 04:18:38.301030 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:38.301005    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-gxb72_f2e46a1b-7b03-4ce0-aa6e-ea5881c0a1d9/insights-operator/1.log"
Oct 30 04:18:38.768413 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:38.768388    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dkfnj_6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da/kube-rbac-proxy/0.log"
Oct 30 04:18:38.876164 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:38.876140    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dkfnj_6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da/exporter/0.log"
Oct 30 04:18:38.986219 ip-10-0-128-241 kubenswrapper[4960]: I1030 04:18:38.986196    4960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dkfnj_6e0c92eb-fdcf-4e6d-bd2a-65a80e2872da/extractor/0.log"